Web server and method for hosting a web page for presenting location based user quality data related to a communication network

ABSTRACT

A computing device forms a predictive composite score by retrieving geographically based wireless networking performance data derived from at least one mobile device. At least some of the geographically based wireless networking performance data is temporally averaged and geostatistically combined to form the predictive composite score. The predictive composite score is a forecast of expected performance quality of a wireless communication network in a particular geographic region.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/801,916, filed Mar. 13, 2013, now allowed, which is acontinuation-in-part of U.S. patent application Ser. No. 13/745,335,filed Jan. 18, 2013, now U.S. Pat. No. 9,083,610, which is a divisionalof U.S. patent application Ser. No. 12/574,655, filed Oct. 6, 2009, nowU.S. Pat. No. 8,379,532, which claims the benefit under 35 U.S.C.§119(e) of U.S. Provisional Patent Application No. 61/103,141 filed Oct.6, 2008, which applications are incorporated herein by reference intheir entirety.

BACKGROUND

1. Technical Field

Embodiments of a web server host a web page that delivers information tousers, the information related to the performance quality of a wirelesscommunication network.

2. Description of the Related Art

Wireless network providers conventionally use relatively complex,expensive, private and inflexible methods for collecting informationabout the quality of their wireless network. More particularly, networkproviders use large vehicles outfitted with special equipment connectedto groups of mobile phones and specialized network monitoring analyzers

The data collected by network providers generally does not reflect atrue user experience. Frequently, the data collected is theoreticaldata. In other cases, instead of true user data, the collected data isat best a representation of the results of particular tests executed bythe test operator. In addition, the test equipment used is specializedfor performing the tests specified by the network provider, and thecollected data is often very complex. The resulting complex testinformation is often not available to the end user or to the generalpublic.

BRIEF SUMMARY

A web-server is taught. The web-server communicates wireless networkingperformance data that reflects the true user experience of a mobiledevice operating in a particular location and on a particular wirelessnetwork. According to one embodiment, the web-server interactivelycommunicates wireless networking performance data. The web-servercommunicates a first set of screen data to a computing device, the firstset of screen data soliciting user input. The web-server receives userinput information from the computing device, and the user inputinformation includes an identified geographical location, an identifiedwireless communication network operator, and at least one keyperformance indicator (KPI) representative of a user experience of thecomputing device operating in the identified geographical location on awireless network operated by the identified wireless communicationnetwork operator.

The web-server retrieves wireless networking performance datacorresponding to the KPI in the identified geographical area on thewireless network. The web-server then communicates a second set ofscreen data having a visual representation of a geographical areacorresponding to the identified geographical location and a third set ofscreen data to overly the visual representation of the geographical areacorresponding to the identified geographical location wherein the thirdset of screen data includes the retrieved wireless networkingperformance data.

According to one embodiment, a web-server method for interactivelycommunicating wireless networking performance data is taught. The methodincludes the step of outputting, with a first computing device to asecond computing device, a first set of screen data. The first set ofscreen data solicits user input. The method also includes the step ofreceiving, at the first computing device from the second computingdevice, user input information in response to the solicitation. The userinput information includes an identified geographic location, anidentified wireless communication network operator, and at least one keyperformance indicator representative of performance quality of a mobiledevice previously operated in the identified geographic location on awireless communication network. The wireless network is controlled bythe identified wireless communication network operator. The method alsoincludes the step of outputting, with the first computing device to thesecond computing device, a second set of screen data. The second set ofscreen data presents a measurement commensurate with the user inputinformation.

According to another embodiment, a computer readable medium has contentthat the web-server to execute the method steps.

According to another method, a web-server device for interactivelycommunicating wireless networking performance data has a centralprocessing unit (CPU) and a memory having software. The software isexecutable by the CPU and operable to execute the method steps.

According to one embodiment, a method of processing wireless networkingperformance data is taught. The method includes the step of retrieving aset of geographically based wireless networking performance data from afirst computing device. The set of geographically based wirelessnetworking performance data is derived from at least one mobile deviceprovisioned for typical operation on a wireless communication network.The method also includes the step of parsing the steps of geographicallybased wireless networking performance data according to a set of keyperformance indicators and calculating a geographically based compositescore representative of wireless networking performance quality from atleast some of the key performance indicators. Finally, the methodincludes the step of formatting the composite score for display on anoutput device.

According to one embodiment, a computing server is taught. The computingserver has a central processing unit (CPU), an input/output port, and amemory associated with the computing server. The memory is configured tostore a software application, and the software application is configuredto receive, via the input/output port, a set of key performanceindicators representative of at least one user experience of a mobiledevice previously operated in an identified geographic location on awireless communication network. The software application is alsoconfigured to calculate with the CPU, a geographically based compositescore representative of wireless networking performance quality from atleast some of the key performance indicators in the set and format thecomposite score for output.

According to one embodiment, a method to form a predictive compositescore is taught. Geographically based wireless networking performancedata is retrieved from a first computing device; the geographicallybased wireless networking performance data is derived from at least onemobile device. At least some of the geographically based wirelessnetworking performance data is temporally averaged and geostatisticallycombined to form the predictive composite score. The predictivecomposite score is a forecast of expected performance quality of awireless communication network in a first geographic region.

According to one embodiment, a non-transitory computer readable mediumincludes content that configures a computing device to perform certainacts. The acts include retrieving a set of geographically based wirelessnetworking performance data derived from at least one mobile computingdevice, parsing the set of geographically based wireless networkingperformance data according to a set of key performance indicators, andgenerating a predictive composite score that represents an estimation ofwireless network performance quality of a wireless network from at leastsome of the key performance indicators. The predictive composite scoreis formulated by mathematically combining a plurality of factors, eachof said plurality of factors associated with a same geographic area

According to one embodiment, a method to predict a performance qualityof a wireless communication network is taught. The method includesassigning boundaries to a plurality of geographic areas, retrieving aset of wireless networking performance data associated with theplurality of geographic areas, the set of wireless networkingperformance data derived from at least one mobile device, spatiallyaggregating the set of wireless networking performance data, andgeostatistically calculating a predictive composite score from thespatially aggregated data, the predictive composite score representativeof wireless networking performance quality in a first geographic area ofthe plurality of geographic areas.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The components in the drawings are not necessarily to scale relative toeach other. Like reference numerals designate corresponding partsthroughout the several views.

FIG. 1 illustrates a data gathering system conventionally used in theprior art to collect communication network performance data.

FIG. 2 illustrates a wireless network performance measurement systemaccording to one embodiment of the invention.

FIG. 3 illustrates a web server in greater detail in another embodimentof the wireless network performance measurement system.

FIG. 3A illustrates examples of the mobile devices of FIG. 2.

FIG. 3B illustrates several components of mobile devices from FIG. 3A.

FIG. 3C illustrates portions of a non-limiting embodiment of a server120 of FIG. 2.

FIG. 4 illustrates one embodiment of dialog boxes presented on a websiteas described herein.

FIG. 5A illustrates an embodiment of a selected geographical mappresented on the website.

FIG. 5B represents a set of coded grids overlying the geographical mapof FIG. 5A.

FIG. 5C illustrates a geographical area having a set of overlying codedgrids.

FIG. 5D illustrates a group of graphs or plots correlating variousfactors that influence network quality in a given geographic area.

FIG. 5E illustrates a set of exemplary results of kriging algorithmsaccording to one embodiment.

FIG. 6 is a flowchart illustrating a process used by a web-serverembodiment.

FIG. 6A is a flowchart illustrating a process used by a mobile deviceembodiment.

FIG. 6B is a flowchart illustrating one embodiment of a process used bythe computing server.

FIG. 6C is a flowchart illustrating processes that may be used byembodiments of the wireless network performance measurement system togenerate a predictive composite score.

DETAILED DESCRIPTION

People use mobile communication devices for business, pleasure, safety,and many other reasons. Many conventional mobile devices are used bothfor voice communications as well as for data communications. Forexample, people use their mobile devices to talk to other people, tointeract with Internet web pages on mobile devices, to send and receivetext messages, to receive geographical positioning information, and manyother things. Accordingly, the quality of the wireless networkconnection and usability of the mobile device are very important topeople.

The people that use mobile communication devices refer to the underlyingtelecommunications network with a wide variety of names. For example,the wireless communication network referenced herein may alternativelybe known as a wireless network, a cellular network, a mobile network,and any combination of same or similar words. The term network as usedherein is intended to broadly encompass any wireless communicationsnetwork.

The operators of the wireless communications network permit some mobiledevices to access and communicate over the wireless communicationsnetwork. Generally, users of mobile devices pay the operators for usingthe wireless communications network. The operators are referenced hereinas network providers, service providers, owners, carriers, and anycombination of same or similar words.

The users of mobile devices that communicate via the wirelesscommunications network include individuals and other computing devices.The mobile devices and users of the mobile devices are interchangeablyreferenced herein as users, mobile devices, clients, cell phones, andany combination of same or similar words.

One quality of the wireless connection is the network signal strength ina particular geographical area. This quality is a factor that directlyaffects the perceived usefulness, efficiency, and overall wirelessnetwork experience provided by the mobile device. In some cases, peopleinconveniently alter their schedules and their locations particularly toimprove their wireless network experience. Accordingly, some mobiledevices have some type of ‘signal strength’ indicator, and wirelessnetwork providers use this to try to differentiate their quality ofservice from that of their competitors.

Other factors that affect a user's perception of quality include batterylife, audio quality, propensity of the device to not drop calls, abilityto make and receive calls in remote and/or densely urban areas,accessibility to data services, technology level of data services, andthe like. In many cases, the user determines the likelihood of beingsatisfied in these factors based on the signal strength indicatorregardless of whether or not the indicator has any relation to thefactors.

Wireless network providers collect data for many reasons. One reason isto generate tangible results to support their ‘quality of service’marketing efforts. The data collected by the network providers istypically complex and not readily available to consumers. In some cases,the data collected by the network providers is merely theoretical, andin other cases, the data is gathered with specialized test equipment.

Wireless network operators usually display their theoretical and/orgathered quality information in simplified form on graphical maps. Thegraphical maps are often only found on the particular network carrier'swebsite. The graphical maps are marketing tools for the network carrier.The graphical maps are often simplified to provide to users anunderstanding or estimate of how the operator's network may perform at aparticular location. In some cases the limited information is merelysignal strength and/or a listing of the type of data connections thatare made available. In other cases, there is even less information.

FIG. 1 illustrates a data gathering system 100 conventionally used tocollect communication network performance data. The system 100 generallycomprises a large vehicle, such as a van or truck, carrying a collectionof test equipment. Several wireless devices 102 are coupled toelectronic data collection devices 104. Respective antennas 106 aretuned in various configurations and also coupled to wireless devices 102or the electronic data collection devices 104. The antennas 106 areoperable to transmit and/or receive communication signals from one ormore wireless communication networks. A power source 108 is used as apower supply for all of the test equipment.

In some cases, the wireless devices 102 are generally available, retailcellular phones, and at least some of the retail cellular phones may beconfigured with specific a software program added for the test operator.In such cases, the software program is custom designed and operable toexecute a particular test desired by the test operator. For example, theprogram may put the cellular phone in a predetermined state, whichfacilitates testing. Even in these cases, however, the mobile deviceshave nevertheless been customized specifically for the industrializedtesting environment of data gathering system 100.

In other cases, the wireless devices 102 are customized wirelessequipment specifically used to facilitate testing. These wirelessdevices may have electronic hardware and software used to transmit knownsignal patterns, receive particular frequencies, and collect specificwireless communication network data.

The power source 108 in the data gathering system 100 is supplied by theon-board vehicle power system and is used to supply power for all of thesystem 100 test equipment. The vehicle's DC power is tapped and isolatedto be a test equipment power source for power source 108. The testequipment power source provides one or more power supplies forindividual pieces of test equipment. The supplies are stepped up or downas necessary and regulated for each of the test equipment devices. Insome cases, one of the individual supplies is further modified from a DCpower source to provide AC power for conventional 110 VAC testequipment.

The purpose of the conventional power source 108 is to provide anunlimited supply of power to satisfy the demands of the test equipment.The conventional data gathering system 100 is not concerned with powersaving or power consumption and so in the conventional data gatheringsystem 100, power measurement factors are not considered. Since powersupply is not one of the variables to be tested, efforts are made tomake it as constant as possible. It is designed in the data gatheringsystem 100 for power source 108 to be an unseen, unnoticed part of thetest equipment.

FIG. 2 illustrates a new wireless network performance measurement system110. Metropolitan areas 112, rural areas 114, and mountainous areas 116,are among the many areas where consumers use wireless communicationdevices. A cellular communication network 122 provides cellular serviceto each of these areas, albeit with widely differing quality. That is,to some degree, every area served by cellular network 122 is subject tolow signal strength, sporadic network outages, poor communicationquality, dropped calls, and a host of other problems broadly identifiedas “low user quality.”

Cellular communication network 122 may be any available network suitablefor transmitting and receiving data between mobile and/or fixed pointdevices, or the cellular network 122 may be dedicated to a primarypurpose of delivering the information to known subscribers. Cellularcommunication network 122 may include both wired and wirelessinfrastructure; for example, transceivers, repeaters, switches,antennas, routers, public switched telephone networks, codecs,multiplexers, de-multiplexers, and the like.

Often, but not always, the hardware used to implement a particularnetwork 122 is dedicated to a particular type of wireless networktechnology. For example, some technologies include Global System forMobile Communications (GSM), General Packet Radio Service (GPRS),Enhanced Data Rates for GSM Evolution (EDGE), Code Division MultipleAccess (CDMA), Universal Mobile Telecommunication System (UMTS) HighSpeed Downlink Packet Access (HSDPA), Evolution-Data Optimized (EV-DO),integrated Digital Enhanced Network (iDEN), Worldwide Interoperabilityfor Microwave Access (WiMax), various 802.11 standards (WiFi),femtocells, satellite-based communications, and many others now or laterdeveloped. These technologies may share components or they may haveinfrastructure dedicated to their type of technology. The combination ofeach of the elements of the wireless network contributes to the qualityor lack of quality of the wireless network 122.

Dispersed throughout the metropolitan, rural, and mountainous areas112-116, a plurality of individual mobile devices 118 a-d are operatedin a manner typified by an average cellular service customerhavingaccess to contracted wireless network services. In some cases, thecellular service customer is an individual user, and in other cases, thecustomer is a business entity that provides the mobile device to arepresentative. In either case, the mobile devices 118 a-d areprovisioned for typical operation on a wireless communication network.

Each of the identified mobile devices 118 a-d of FIG. 2 is outfittedwith a software program used to collect selected cellular network,geographical, and device test data. Data collection by the mobiledevices 118 a-d is directed by a server 120 in a central office.

The central office may have one or more servers 120. In some cases,several servers 120 are used to provide redundant operations, whichimprove the robustness of the data gathering operation. In some cases,several servers 120 are used to communicate with a large number ofmobile devices 118 a-d. Additionally, the one or more central officesmay be located in dense, urban locations or in remote, rural locations.

Server 120 sends wireless communication instructions to the mobiledevices 118 a-d directing each device to collect and transmit test databack to the central office server 120. Embodiments of mobile devices 118a-d are disclosed in a corresponding U.S. Provisional Patent Applicationbearing No. 61/103,174, filed Oct. 6, 2008, and a corresponding U.S.Utility patent application, filed Oct. 6, 2009, each entitled “MOBILEDEVICE AND METHOD FOR COLLECTING LOCATION BASED USER QUALITY DATA,” andeach naming each of the inventors of the present application, filedconcurrently and incorporated by reference in their entirety.

Server 120 sends wireless communication instructions to the mobiledevices 118 a-d directing each device to collect and transmit test databack to the central office server 120. Embodiments of server 120 aredisclosed in a U.S. Provisional Patent Application bearing No.61/103,160, filed Oct. 6, 2008, and a corresponding U.S. Utility patentapplication, filed Oct. 6, 2009, each entitled “SERVER DEVICE AND METHODFOR DIRECTING MOBILE DEVICES TO COLLECT AND COMMUNICATE LOCATION BASEDUSER QUALITY DATA,” and each naming each of the inventors of the presentapplication, filed concurrently and incorporated by reference in theirentirety.

FIG. 3 illustrates web server 126 in greater detail in anotherembodiment of the wireless network performance measurement system 110.Web server 126 has operative electronic circuitry including a centralprocessing unit 128 (CPU), multiple input/output (I/O) ports 130, a highbandwidth network connectivity port 132, volatile memory 134 (RAM), andnon-volatile memory 136. Memory 136 is any non-volatilecomputer-readable media including, for example, flash memory, magneticmedia such as a hard-disk, an optical disk drive, a floppy diskette, aCD-ROM, and/or the like.

As known by one skilled in the art, the memories 134, 136 comprise anycombination of volatile and non-volatile computer-readable media forreading and writing. In some cases, a particular memory is separatedvirtually or physically into separate areas, such as a first memory, asecond memory, a third memory, etc. In these cases, it is understoodthat the different divisions of memory may be in different devices orembodied in a single memory.

In a preferred embodiment of FIG. 3, within the memory 136, severalsoftware programs are resident including an operating system 138, a webserver application 140, a wireless network performance softwareapplication 142, and an intermediary networking software application144.

The CPU 128 of the web server 126 retrieves and executes instructionsfrom memory 136 and/or RAM 134. The operating system 138 includesapplication and driver software that permits additional applicationsoftware and system administrators to control the operation of server126. For example, particular applications and drivers are used to acceptuser input and to provide system output through I/O ports 130. That is,keypads, computer mice, memory cards, serial ports, bio-sensor readers,touch screens, and the like are all useful to a system administrator orother web-server operator to provide control information into the webserver 126. Displays, printers, memory cards, LED indicators, speakers,and the like are all useful to present output information to theweb-server operator.

In addition to application and driver software within the operatingsystem 138, operating system 138 hosts a web server 140 application. Inone embodiment, the web server application 140 is a Microsoft InternetInformation Services web server. The web server application hostsparticular static and dynamic web pages (e.g., screen data) such thatusers of the Internet are capable of interactively viewing. The wirelessnetworking performance application 142 described herein includes webpages hosted by the web server application 140.

The intermediary networking software application 144 is a collection ofone or more software programs that provide connectivity to computingresources outside of the web-server 126. For example, conventionaltransmission control protocol/Internet protocol (TCP/IP), file transferprotocol (FTP), user datagram protocol (UDP), and the like are useful topermit data such as web pages to be served to the outside computingresources, and further are useful to permit users of the outsidecomputing resources to input data into the web server 126. That is, theoperating system 138 and web server application 140 are bidirectionallycoupled to the intermediary networking software application 144 and theintermediary networking software application 144 is bi-directionallycoupled to individual circuits of the high bandwidth networkconnectivity port 132. The bidirectional coupling permits the web serverapplication to solicit a user for input information, accept the userinformation, and output wireless networking performance data as screendata, screen updates, audio data or the like, commensurate with the userinput information.

The high bandwidth network connectivity port 132 includes electronicnetworking circuits such as a Digital Signal 1 (T1), Digital Signal 3(T3), fiber-optic, Ethernet, fax/modem, and other telecommunicationscircuits. The telecommunications circuits of the high bandwidth networkconnectivity port 132 provide the hardware layer interconnectivity ofthe web server 126, through a telecommunications network 146 to a widerange of outside computing resources.

The telecommunications network 146, which includes the cellularcommunication network 122, further includes conventional cable networks,satellite networks, fiber-optic networks, and the like. In a preferredembodiment, the telecommunications network 146 includes anycommunication hardware and software that cooperatively works to permitusers of computing devices to view and interact with Internet web pages.

The outside computing resources include mobile devices 118 a-d,conventional user computers 148, and even server 120. In a preferredembodiment, mobile devices 118 a-d are directed by server 120 to collectreal-time network performance data as the voice and computer datatravels from and between other mobile devices, other web servers, andother user computers through the cellular network 122. The networkperformance data collected by the mobile devices 118 a-d is communicatedto server 120, where it is processed. The processed data is communicatedby server 120 to web server 126. The web server 126 interactivelypresents the data to users of the mobile device 118 a-d, other mobiledevices, and user computers 148.

User computers 148 comprise one or more computing devices operatingseparately or in a networked environment. In some cases, user computer148 is a conventional personal computer operated by a single user. Inother embodiments, user computers 148 are network computers processingdata for other industrial, commercial, or business purposes.

Resident in memory 136, the wireless networking performance application142 embodies the software code of a wireless networking performancewebsite 150. Generally speaking, the website 150 displays collectedlocation based user quality cell phone performance metrics to users viathe Internet. That is, the website 150 and the wireless networkingperformance application 142 work cooperatively to provide detailed,geographic location-based wireless network information and userexperience quality data in a manner that compares the information acrossmultiple, different cellular network operators and multiple, differentmobile devices. The user experience quality data that is presented onthe website 150 is useful to users of the website 150 during theirevaluation and comparisons of particular wireless cellular networks andparticular mobile devices.

The user-experience quality data presented on website 150 includeshelpful information regarding both voice communications and datacommunications of particular cellular networks and particular mobiledevices. The data represents accurate information about end-userwireless network experiences, which has been collected from specificmobile devices 118 a-d. The specific mobile devices 118 a-d are runningan application which accesses the wireless network information. Thespecific mobile devices 118 a-d collect, process, and communicate theinformation, which is then presented on website 150 as key performanceindicators (KPIs) representative of performance quality of the mobiledevices 118 a-d as they were previously operated in identifiedgeographic location on the particular cellular networks.

Website 150 serves many purposes. For example website 150 presentsrealistic and accurate end user quality network KPI information relatedto each network cellular carrier that provides cellular network servicein a particular geographic location. As another example, website 150operates as a centralized website where consumers can access detailedinformation that pertains to mobile devices such as usage patterns,voice quality, battery life, dropped or interrupted connections, and thelike. As another example, since website 150 provides so much informationto consumers, a significant number of consumers will access website 150.And, since a significant number of consumers will access website 150,goods and services providers will pay to provide additional usefulinformation such as advertising to the consumers.

Website 150 prominently displays information about the quality of thesupported wireless cellular networks. A supported wireless cellularnetwork is one from which mobile devices 118 a-d collect data, theresults of the data collection being presented on website 150. Thedisplayed information reflects actual user experience on the particularwireless cellular network for a particular geographic location.

Many types of information related to a particular wireless cellularnetwork are made available to consumers. Consumers are able to viewcomparative information from many wireless cellular networks on website150. In addition, consumers are able to view comparative informationfrom many different mobile device models. For example, some keyperformance indicators that are viewable on website 150 include networksignal strength, upload data transfer speed, download data transferspeed, data latency, video quality, dropped calls, error messages, datatechnology transition, audio quality, and others.

In a preferred embodiment, users of website 150 wish to view the KPIdetails for a particular wireless cellular network in a particularlocation. The website 150 presents a first dialog box to the user, andthe user enters a zip code or other location identifier such as a streetaddress, state, or metro/city name. Website 150 will present informationrelated to the location selected by the user. For example, theinformation presented may include the number of mobile devicescollecting data in the geographic area, the times and dates when data iscollected, the size, shape, elevation, urban density, and other factorsrelated to the geographic area, the number, identity, and location ofcell towers used by the particular wireless cellular network and theiroperational state, and other information. In some cases, website 150 mayeven show only the data collected by a particular identified mobiledevice 118 a-d.

After information related to the selected area of interest is displayed,website 150 presents a second dialog box to the user. The second dialogbox presents an opportunity for the user to select various wirelesscellular network operators for which the user would like to view KPIdetails. Through a third dialog box presented by website 150, the userhas an opportunity to select one or more KPIs for which website 150 willpresent specific data.

FIG. 3A illustrates examples of the mobile devices 118 a-d of FIG. 2. Ahandheld global positioning device 118 a receives positioninginformation from a satellite network 122 a and mapping information froma cellular network 122. A laptop computer 118 b is outfitted with anexternal wireless card for sending and receiving voice and/or data overa cellular network 122. A mobile telephone 118 c is most often used forvoice and data communications on a cellular network 122. An automobile118 d commonly has cellular network 122 accessing capability for voiceand data communications. The representative illustrations of FIG. 3A aremerely non-limiting examples of the type of mobile wireless networkclient devices useful for collecting wireless network performancemeasurements. Many other client devices, including personal mediaplayers (PMP's), personal digital assistants (PDA's), handheld personalcomputers (HPC's), and the like can also be configured as describedherein to collect network data. In fact, virtually any computing devicewith a compatible transceiver and proper configuration can be a mobiledevice 118 a-d.

Also illustrated in FIG. 3A are a particular collection of softwareinstructions 124. The software instructions 124, as described herein,are useful to direct the mobile devices 118 a-d in collectingcommunication network performance quality data. Software instructionsalso have a configuration data file 124 a, which is useful for directingthe mobile device 118 a-d data collection operations.

FIG. 3B illustrates several components that mobile devices 118 a-d havein common. The mobile devices 118 a-d are electronic communicationsdevices having a central processing unit 128 a and a volatile and/ornon-volatile memory 130 a. The memory 130 a stores instructions and dataacted on by CPU 128 a.

Memory 130 a comprises any combination of volatile and non-volatilecomputer-readable media for reading and writing. Volatilecomputer-readable media includes, for example, random access memory(RAM). Non-volatile computer-readable media includes, for example, readonly memory (ROM), magnetic media such as a hard-disk, an optical diskdrive, a floppy diskette, a flash memory device, a CD-ROM, and/or thelike.

Inside memory 130 a, a particular collection of software instructions124 are executed by CPU 128 a in order to collect performancecharacteristics of the mobile device 118 a-d, e.g., network data. Thesoftware instructions 124 operatively configure hardware and othersoftware in the mobile device 118 a-d so that particular test conditionsmay be set up and particular measurements may be taken. The testconditions, the resulting measurements, and other data are stored in afile system 132 a. File system 132 a may be a traditional file system,one or more database systems, or any other method known in the art forstoring such information, or any combination of the above, includingimplementation using distributed computing techniques.

The mobile devices 118 a-d further have a collection of input/outputports 134 a for passing data over short distances to and from the mobiledevice 118 a-d. For example, serial ports, USB ports, WiFi ports,Bluetooth ports, IEEE 1394 FireWire, and the like can communicativelycouple the mobile device 118 a-d to other computing apparatuses.

Mobile devices have a battery 136 a and a display 138 a. In many mobiledevices 118 a-d, a signal strength indicator 140 a is prominentlypositioned on the display 138 a to provide network communicationconnectivity information to the mobile device 118 a-d operator.

A cellular transceiver 142 a is used to couple the mobile device 118 a-dto other communication devices through the cellular communicationnetwork 122. In some cases, software 124 and data in file system 132 aare communicated between the mobile device 118 a-d and a server 120 viathe cellular transceiver 142 a. That is, bidirectional communicationbetween a mobile device 118 a-d and a server 120 is facilitated by thecellular transceiver 142 a. For example, a server 120 may download a newor updated version of software 124 to the mobile device 118 a-d over thecellular communication network 122. As another example, the mobiledevice 118 a-d may communicate network performance measurement datastored in file system 132 a to the server 120 over the cellularcommunication network 122.

Each mobile device 118 a-d client has electronic memory 130 a accessibleby at least one central processing unit 128 a within the device. Thememory 130 a is programmed with software 124 that directs the one ormore central processing units 128 a. Some of the software modules in thememory 130 a control the operation of the device 118 a-d with respect tocollection of data. In some cases, software 124 directs the collectionof individual datums, and in other cases, software 124 directs thecollection of sets of predetermined cellular network, geographical, anddevice test measurements.

Software 124 may include a fully executable software program, a simpleconfiguration data file 124 a, a link to additional directions, or anycombination of known software types. When the server 120 updatessoftware 124, the update may be small or large. For example, in somecases, server 120 downloads a small configuration data file 124 a to aspart of software 124, and in other cases, server 120 completely replacesall of the present software 124 on the mobile device 118 a-d with afresh version. In some cases, software 124 is encrypted, encoded, and/orotherwise compressed for reasons that include security, privacy, datatransfer speed, data cost, or the like.

A mobile device 118 a-d executing software 124 addresses manyshortcomings of the conventional methodology. Generally speaking, theconventional methodology uses an optimized antenna and transceiversystem, which is not capable of collecting true, end-user experiencedata. In contrast, software 124 enables a specific mobile device 118a-d, provisioned for typical operation on a wireless communicationnetwork, which a user operates to access the wireless network, tocollect location based true user quality data. The mobile device 118 a-dis operable to access contracted wireless network services provided bythe wireless cellular communication network 122.

More particularly, software 124 provides many advantageous features forthe wireless network performance measurement system 110. Theadvantageous features are not found in the conventional methodology. Forexample, software 124 runs as a software application on aproduction-type mobile device 118 a-d whereas the conventionalmethodology requires customized test equipment. Software 124 minimizespower consumption on the mobile device 118 a-d while collectinginformation such as network parametric data, device location data, anduser experience quality data, but the conventional methodology ignorespower consumption. In addition, software 124 is directed by acentralized server 120, and software 124 sends data to a centralizedserver farm, but the conventional methodology has no such provision. Thedata sent by the mobile device 118 a-d to the centralized server 120 canbe used to evaluate, compare and improve the overall user experience onthe wireless network that a user of device 118 a-d operates for bothvoice and data communications.

In a preferred embodiment, software 124 collects information about thequality of the wireless communication network 122 that reflects actualuser experience. The technique used in a mobile device 118 a-d is toemploy the actual hardware that a user operates when accessing thewireless network 122. The actual hardware includes the radio module, theaudio module, the display module, and the other circuitry of the mobiledevice 118 a-d.

One information datum collected includes network signal strength.Network signal strength is collected using the same network signalstrength measurement that is presented to the user. This is usefulbecause a user often monitors the network signal strength, e.g. throughan icon. It is understood that the true network quality is notnecessarily reflected correctly by the presentation to the user;however, the network signal strength presented to the user affects theuser's impression of the network quality.

Network signal strength may have many components and representations.That is, network signal strength is particular to the mobile device 118a-d and the network 122 where the device is operating. For example, insome cases, network signal strength is a simple power measurement inmilliwatts (mW) of the magnitude of the received signal's electricfield, and in other cases, the power is measured as a ratio in decibels(dB) related to a one milliwatt reference signal (dBm). In still othercases, network signal strength refers to a signal to noise ratio, whichis a relative determination between the strength of the desired radioinformation signal and the undesired noise interfering with it. Broadlyunderstood, network signal strength is any measurement thatcharacterizes the intensity and integrity of the wireless informationsignal received by the mobile device 118 a-d.

Another information datum collected is data transmission speed. Datatransmission speeds are generally collected using the highest level ofapplication programming. This datum is useful because it takes intoaccount even the degradation due to the mobile device hardware,software, and current operational environment. Data transmission speedsmay include both voice or other data. Data transmission speedmeasurements may include both transmit and receive measurements, i.e.,data transmitted by the mobile device 118 a-d through the network 122and data received by the mobile device 118 a-d through the network 122.In some cases, the data transmission speeds are known as data downloadand data upload speeds.

Audio quality information is also collected. Audio quality is measuredusing the actual microphone and speaker of the mobile device 118 a-dthat the user uses to make and receive calls. In this way, informationrelated to a user's true audio experience will be measured.

In the preferred embodiment of the new wireless network performancemeasurement system 110, several considerations are taken into account tominimize the impact of data collection on the user's actual experienceof operating the mobile device. For example, power management, systemresponsiveness, audio path disruption, ease of installation, andbackground execution are all considered as described herein. If thefunctionality of any of these considerations is degraded by the datacollection, then the user's experience is adversely affectedsignificantly. The data collection software is selected to execute in amanner that will not affect the measurement of the system beingmeasured. Several examples of this will now be given.

A first example is power consumption. Even though mobile devices 118 a-dare capable of hard-wired, external power or conventional alternatingcurrent (AC) power connections, many devices are powered substantiallyby batteries. In some cases, the batteries are disposable, and in othercases, the batteries are rechargeable, but in either case, batteries aregenerally capable of operating the device for only a short time beforeneeding to be replaced or re-charged by the device operator. That is,after just a few hours, days, weeks, or some other short time period,the operator must attend to the expended batteries. Accordingly,maximizing the battery life of a mobile device 118 a-d is often a goalof device manufactures, device sellers, and device users.

The mobile device 118 a-d uses power during data collection. That is,the execution of software and the operation of hardware solely for thepurpose of collecting, processing, and distributing test data willexpend power that would not otherwise be expended. If wireless networkperformance measurement system 110 minimizes its power usage, then theattention to the battery required by the device 118 a-d is minimized.The additional power consumed by the testing can be considered and takeninto account when measuring power consumption in one embodiment. Thepower consumed by the testing software 124 will not be considered apower consumed by the system in such an embodiment. In otherembodiments, the battery life will be reported based on the life as itwould have been without the test software 124 running at various times.

In a preferred embodiment of software 124, data is collected only incertain locations and as infrequently as possible to minimize overallpower consumption of the mobile device 118 a-d. By reducing the datacollection locations and frequency, power usage is reduced andcorrespondingly, the user's perception that software 124 isdetrimentally affecting the power usage of the mobile device 118 a-d isreduced.

A second example is internal CPU usage. The test software 124 makes useof the CPU, so that when it is executing, the CPU usage goes higher. Thesoftware 124 monitors how much its own use is adding to the CPU usagefactor and subtracts this from the measurement, according to oneembodiment.

A third example is bus throughput on internal bus data transmissionspeed. To the extent running the tests affects the internal busthroughput, this is accounted for and removed from the test results.

A fourth example is interrupt measurements, which, since the CPUreceives interrupts as part of its standard operations and also whenbeing tested by software 124, these test based interrupts should not becounted in the system performance. More or less of these accommodationsof the measurement affecting the measured quantity can be considered andtaken into account by the software 124 according to various embodiments.

In order to collect data only in certain locations, one embodiment ofsoftware 124 records and updates the name and/or ID of the currentlyconnected (i.e., associated) cell tower. In many devices, the radiomodule independently communicates with one or more cell towers. A celltower that has established connection with a device supplies its nameand/or ID to the device, and the name and/or ID is stored in the radiomodule. The data exchange that transfers the cell tower name to themobile device 118 a-d is a transaction that is executed independently onthe device. The transaction is even executed when the mobile device 118a-d is in a power saving mode.

The wireless network performance measurement system 110 uses the celltower name to break down data collection coverage areas into sectors.Server 120 maps out these sectors to determine which sectors should beupdated with fresh data. Based on the sector mapping, or based on theleast recently updated sector, or based on some other means, server 120will send a particular cell tower update list to the mobile device 118a-d. The cell tower update list includes cell tower names identifyingsectors where server 120 requests updated wireless network performancedata. When the mobile device 118 a-d detects that it is connected to acell tower on the list, the mobile device 118 a-d will collect datawhenever the device moves more than some predetermined amount, forexample 30 meters.

In another preferred embodiment, a unique user location profile may begenerated for a mobile device 118 a-d operating in a wireless networkperformance measurement system 110. For example, each mobile deviceidentified by server 120 may be given a unique identifier. The uniqueidentifier may be random number, an Electronic Serial Number (ESN), anInternational Mobile Equipment Identifier (IMEI), or some other number.In many cases, the number is encrypted. After generating the uniqueidentifier, the identifier may be combined with typical geographic andtime information to create the unique user location profile for eachmobile device 118 a-d.

After a user location profile is generated for a mobile device 118 a-d,the profile is available for use by server 120 to collect particulardata at particular times and in particular locations. In some cases,server 120 can use the profile in a predictive manner, and in othercases, server 120 can use the profile to make decisions about dataretrieval from particular mobile devices 118 a-d. The server 120 sendsnew software 124 (e.g., a configuration data file 124 a) to the mobiledevices 118 a-d in order to change the testing being performed. It alsosends instructions and queries to the software 124 to cause it toexecute selected routines as directed by the server 120 or for theserver to understand what version of the software 124 is present on eachof the devices 118 a-d, what tests each is conducting at any given time,and to learn other things about the respective devices 118 a-d.

In order to collect data only in certain locations and as infrequentlyas possible, server 120 actively monitors how many mobile devices 118a-d are currently in a particular sector. Server 120 further monitorswhich sectors have outdated or otherwise insufficient data. Server 120may increase or decrease the frequency of data collection of a mobiledevice 118 a-d based on the age, quality, or volume of data previouslygathered. Software 124 in a mobile device 118 a-d is responsive to therequests for data of server 120.

When the wireless network performance measurement system 110 isinitialized in a new geographical area, the number of data collectionmobile devices 118 a-d may be small. Performance and powerconsiderations are considered for each mobile device 118 a-d, so server120 may only request data in certain areas of interest, and server 120may function to prevent or minimize the collection of data inoverlapping areas. For example, in areas where there are few mobiledevice clients 118 a-d, the mobile device client may be requested tocollect data with greater frequency only in areas where server 120 doesnot currently have data. Alternatively, server 120 may reduce or suspendrequests for data in areas where server 120 does not currently havedata. In other cases, server 120 may request data based on one set ofparameters, but software 124 of mobile device 118 a-d may decide locallywhether and how to respond to the request based on a local set ofparameters. Parameters that software 124 may use to decide whether andhow to respond include current battery life, current processorutilization, scheduled events, and the like.

In some cases, the wireless network performance measurement system 110is initialized or operating in a geographical area where the number ofdata collection mobile devices 118 a-d is large, e.g., 5000 or moredevices. In such cases, server 120 may further function to adjust thefrequency and/or volume of data collection by a particular mobile device118 a-d. For example, rather than collecting data from every device,server 120 may collect data from only some devices and use techniquessuch as statistical modeling, to create a complete set of wirelessnetwork performance data. As another example, server 120 may direct somemobile devices 118 a-d to collect data of one type and direct othermobile devices 118 a-d to collect data of another type. The wide rangeof data collection directions provided by server 120 in a geographicalarea having many mobile devices 118 a-d reduces the demand for power dueto data collection on each mobile device 118 a-d.

Another technique used by the server 120 to direct data collection onlyin certain locations and as infrequently as possible is to takeadvantage of location based services on the mobile device, for example,the global positioning systems (GPS) within some mobile devices 118 a-d.The GPS is capable of providing latitude and longitude coordinates aswell as other distance and positional information. Operating the GPScircuitry, however, uses a large amount of power on most devices.Accordingly, it is often desired to collect GPS data only when mobiledevice 118 a-d has changed location.

One technique used to minimize the frequency of GPS driven datacollection is to take advantage of the cellular network's capability toprovide the distance of the mobile device 118 a-d from the closestcellular tower or any other cellular tower. The cellular towerinformation is collected by the radio hardware in the mobile device 118a-d, so a request from software 124 to the radio hardware for thisinformation will generally not increase power consumption. Moreparticularly, each radio module on the mobile device 118 a-d monitorsthe distance between the device and the currently connected cell tower.This distance data is independently updated by the radio module on themobile device 118 a-d even in a power saving mode. By monitoring thevalue of this distance, software 124 can detect when the mobile device118 a-d is in motion and when the mobile device 118 a-d has moved from aparticular position by a predetermined amount, e.g., 30 meters.

Using the distance information from the cellular tower, a set of datacollection rules provided by server 120 or otherwise available tosoftware 124 direct whether or not the GPS module needs to be engaged.For example, if the distance to the closest cellular tower has notappreciably changed, then the mobile device 118 a-d may not be directedto capture new GPS coordinates and new network performance data.

Another data set collected by the wireless network performancemeasurement system 110 is the speed of data transmission across thewireless network 122 and within the mobile device 118 a-d. Thecollection of data speed measurements, however, consumes power. Dataspeed measurements involve sending and receiving network data using themobile device's 118 a-d wireless radio transmission and receptioncircuitry.

There are several techniques used to reduce the power consumption of thedata speed measurement test. For example, in one technique, mobiledevice 118 a-d is only directed to collect data when the device is at aspecific location. In a second technique, data speed measurements areperformed infrequently on any given device, but the data speedmeasurements are performed on many devices so that a wide range of datais gathered. In a third technique, data speed measurements are collectedby evaluating user or network directed data transmissions instead ofsending and receiving dedicated test data for the data speedmeasurement. In a fourth technique, dedicated test data is piggybackedon a preexisting data transmission. In many cases, power consumption fordata speed measurement is reduced because the radio is already beingoperated for another purpose, e.g., when a user is checking email,surfing the web, transferring files, or some other user requestedpurpose.

Data collected by the mobile device 118 a-d is returned to server 120.Another technique useful to saving power on the mobile device 118 a-d isto adjust the manner in which the data is returned to server 120.

Many devices, such as mobile device 118 a-d, contain a large amount offile storage space. This storage space is useful to software 124 forstoring wireless network performance data. In one embodiment, thesoftware 124 is designed to send GPS and other test information to theserver 120 only when it is plugged into a power supply. With filestorage capability, software 124 can wait to send the data to the server120 until the mobile device 118 a-d is being charged or is otherwiseoperating from a power source more robust than a battery, such as anexternal source. Alternatively, in cases where it is not reasonable towait until the device is being charged, software 124 can piggyback theperformance data on user initiated data transfers. The piggy-backingtechnique minimizes the need to establish a new connection, which canreduce power consumption.

In some embodiments, software 124 operates in a Microsoft Windows Mobileenvironment. In a Microsoft Windows Mobile environment, the mobiledevice 118 a-d is often connected to a PC using an ACTIVESYNC protocol.The ACTIVESYNC protocol is one of many similar configuration andnetworking protocols that use a wireless or wired network resource totransmit and receive data. In such cases, the mobile device 118 a-d ischarging, however the cellular radio functionality is temporarilydisabled. In these environments, the technique of piggy-backing data onuser-initiated data transmissions is useful to reduce power consumptionon the mobile device 118 a-d.

In other embodiments, software 124 functions in another commerciallyavailable operating system environment. In these cases, software 124 maybe ported to specifically function in the particular operating system orsoftware 124 may be generic enough to operating on several operatingsystems. In these cases, different techniques may be used to save power,transfer information, or the like.

Another useful technique is to take further advantage of circumstanceswhere mobile device 118 a-d is being charged or is otherwise operatingfrom a robust power source (e.g., an external power source). In thesecases, software 124 may be directed to conduct additional or more dataintensive testing. For example, in some cases, mobile device 118 a-d isplugged into a vehicle's power source. If the vehicle, e.g., anautomobile, is moving, then the mobile device 118 a-d may perform anexpanded range of wireless network performance data collection.

Data may also be collected in mobile devices 118 a-d without GPScapability and where the device stays within range of a particularcellular tower name.

The operation of software 124 is preferably un-noticed by a user of themobile device 118 a-d. More particularly, the software 124 should notcause the user to notice an impact on the CPU performance and filesystem of the mobile device 118 a-d. One way to avoid this is toschedule software 124 to run at a very low priority. Another way is topermit user requested actions to be performed without delay. Forexample, in addition to operating at a low priority, software 124 can beconfigured to immediately suspend operation when a user requested actionis detected.

Audio quality data is collected by the wireless network performancemeasurement system 110. Preferably, the measurement of audio dataquality will not be noticed by the user of the mobile device 118 a-d.One technique useful for keeping software 124 from impacting, adverselyand significantly, the user experience of the mobile device 118 a-d isto collect data when the audio circuitry is used, but to suspendprocessing of the data until the audio circuitry is no longer beingused. For example, data may be collected during a phone call or duringaudio playback of a particular file, but any post-processing of the dataoccurs at a later time, and the results collected are transmitted toserver 120 at a later time.

In a preferred embodiment, software 124 is easy to install and theinstallation procedure will have little or no user intervention. Forexample, the installation of software 124 may be composed of a simpleover-the-air (OTA) download having a click install, no questionspresented to the user, and not requiring the user to reboot the device.In this manner, the user will be aware that software 124 has beeninstalled, but the installation procedure will have been relativelyquick and easy.

Also in a preferred embodiment, software 124 will run generally in thebackground, transparent to the user. Preferably, the user will be awarethat software 124 is performing data collection work on the mobiledevice 118 a-d, but the actual operation of software 124 will beun-noticed. In this manner, a user may be encouraged to install software124 and permit the mobile device 118 a-d to collect data for thewireless network performance measurement system 110.

The wireless network performance measurement system 110 presents astructured device management architecture in which to operate software124. The structured architecture permits an external device, server 120for example, to direct the operation of mobile device 118 a-d. Thestructured architecture further permits the data collection of mobiledevice 118 a-d to be synchronized with other mobile devices.

For example, synchronization of data collection permits multiple mobiledevices to be directed with either the same or complementary datacollection techniques. In this manner, the effectiveness of each mobiledevice within the wireless network performance measurement system 110may be increased. For example, in some cases, the real time location ofthe device is used to determine which data will be collected. In othercases, different criteria are used to direct the data collectionincluding the type of device, the current application load, the signalstrength indication, or the like.

Within the wireless network performance measurement system 110, a mobiledevice 118 a-d can be associated with one or more particular groups onthe server 120. The grouping permits a server 120 to configure multiplemobile devices 118 a-d with a common set of data collection rules suchas a data collection update schedule. Generally, the mobile device 118a-d is unaware of which group it is currently associated with, but insome cases, during system testing for example, the mobile device 118 a-dis aware of its group assignment.

Mobile device 118 a-d is operable to accept direction from server 120when server 120 communicates with the mobile device 118 a-d. In oneembodiment, communication between the server 120 and the mobile device118 a-d is facilitated through one or more data sockets. That is, mobiledevice 118 a-d opens a data socket to server 120, which permitsdirection from server 120 when the rules are to be changed, software isto be updated, or any other control information is to be exchanged. Incases where a single socket is opened, the communication of all controlinformation over the single socket is helpful to reduce data networkusage.

Server 120 is operable to update and modify the data collection rulesfor the mobile device 118 a-d. For example, in some cases, server 120configures the mobile device 118 a-d with parameters including a list ofdata points to collect, a list of real-time locations identifying wheredata will be collected, and an indication of how often data pointsshould be collected. Generally, server 120 configures the mobile device118 a-d asynchronously and at any time.

In a preferred embodiment, software 124 includes a configuration datafile 124 a. The configuration data file 124 a is generally received fromserver 120 in a wireless manner, but other techniques may also be used.For example, a wired connection or a memory card may be used to providethe configuration data file 124 a of software 124 of the mobile device118 a-d. Alternatively, the configuration data file 124 a may not evenreside on the mobile device, but instead be remotely located and merelyaccessible by the mobile device 118 a-d.

The configuration data file 124 a of software 124 may be small or large,but typically represents rules passed to the mobile device 118 a-d thatare to be followed by the device when collecting data. In some cases,the configuration data file 124 a has executable code that carries outor otherwise directs the rules, but in other cases, the configurationdata file 124 a has only indicators of which pre-programmed rules thatthe mobile device 118 a-d will follow.

Several data collection rules direct the data collection operations ofthe mobile device 118 a-d. By operating the mobile device 118 a-d undera set of changeable rules, the value of the data that is collected andpassed to the server 120 is improved. For example, in some cases, therules are directed to all mobile devices 118 a-d, and in other cases,the rules are directed only to particular devices. Other preferredembodiments take advantage of the user location profile for theparticular mobile device 118 a-d. That is, based on the uniqueidentifier for the mobile device 118 a-d, the time, location, or otherparameters of the device may be predictively and/or dynamically used todirect the data collection.

One rule passed to mobile device 118 a-d is a cell tower list rule. Thecell tower list rule includes a list of cell tower areas where datacollection is desirable. According to this rule, software 124 monitorsthe identity of a currently connected cell tower, and when the mobiledevice 118 a-d connects to a new cell tower, software 124 scans the celltower list to determine if data collection has been directed. In somecases, the cell tower list is disabled, which is a direction to themobile device 118 a-d to collect data in all areas.

Two other rules passed to mobile device 118 a-d are a data speed testfrequency rule and a data latency test frequency rule. The rules includea value of how many seconds the mobile device 118 a-d will wait betweeneach data speed test or each data latency test respectively. Server 120can direct mobile device 118 a-d to run a data speed or data latencytest at a single frequency, at a particular pattern of frequencies, arandom frequency, or at some other interval. Further, individual celltowers or groups of cell towers in the cell tower list rule cancorrelate independent values for the data speed and data latency testfrequency rules. In addition, when the cell tower list is disabled,independent default values may be specified for the data speed and datalatency test frequency rules.

Through the data sockets, a particular command channel can be opened upfrom server 120 to mobile device 118 a-d. The command channel may beinitialized by either the server 120 or the mobile device 118 a-d, andeither unit may initiate a control cycle. The command channel permitsthe communication of control information between the server 120 and themobile device 118 a-d.

Using the command channel between the server 120 and the mobile device118 a-d, the centralized server 120 can interrogate the software 124 forits current version. For example, server 120 may send a “VERSION”command to mobile device 118 a-d. Software 124 will return its currentversion number to server 120. In one embodiment, the version number willbe a single floating point number.

Using the command channel between the server 120 and the mobile device118 a-d, the centralized server 120 can update the software 124 of themobile device 118 a-d to latest version. Generally, after the software124 has been updated, the new version of software 124 will restartwithout the need for a reboot of the mobile device 118 a-d.

The command channel between the server 120 and the mobile device 118 a-dcan be used for many other commands. For example, an UPDATE command isuseful for sending a name string from the server 120 to the mobiledevice 118 a-d. A TIME_SYNC command is useful for synchronizing thecurrent time, date, day or the like between the server 120 and themobile device 118 a-d.

The use of particular rules for data collection, along with the commandchannel for communicating command information, permits the mobile device118 a-d to collect significant data related to the wireless networkperformance measurement system 110. For example, Table 1 lists severaldata sets and individual data points that are collected by a mobiledevice 118 a-d in a preferred embodiment.

TABLE 1 Data collected and communicated to a server Cellular NetworkSignal Strength 802.11 Network Signal Strength Network Capability level(Ex. 2G, 3GPP, LTE, etc.) Network Technology (Ex. WiMAX, EDGE, etc.)Data Transfer Speed, both Upload and Download Data Transfer Latency fromthe device to the server and the gateway Video Quality at differentbitrates Dropped Call/Network errors Transition from one cellularnetwork technology to another technology such as a transition from 3G toEdge. Audio Quality both directions on different cellular networktechnologies such as 3G, Edge, etc. Battery Life on different mobilephones and different cellular network technologies Lost audio networkpackets that is experienced as choppy or poor audio SMS/Text latencyTower ID Signal to Noise Ratio (SNR)

In a preferred embodiment of a mobile device 118 a-d that is running theinventive data collection operations described herein, a program insoftware 124 executes. In one embodiment, descriptively called theService Mode, the software 124 is operating in the background, with noindication to the user of the mobile device 118 a-d that the program isrunning. At other times, descriptively called the Application Mode, thebackground application is accessible to the user of the mobile device118 a-d to either provide information to the user or to operateaccording to instructions from the user. Embodiments of both the ServiceMode background application and the Application Mode foregroundapplication will now be described with reference to the figures.

For example, on a mobile device 118 a-d, in software 124, a datacollection client software program is initialized and begins executionin the Service Mode. The program requests particular resources from theoperating system that is executing on the mobile device 118 a-d, and theprogram sets up particular data structures in software 124 and filesystem 132, both of which are resident in memory 130 in the preferredembodiment.

In the Service Mode, the program typically has two main functions. Afirst function of the Service Mode operates to collect data from themobile device 118 a-d. A second function of the Service Mode operates tocommunicate the collected data to a central collection server.

In the Service Mode, the user does not have to dynamically interact withthe software 124 to provoke the application to collect the requireddata. For example, a predefined set of rules may be present in software124, or alternatively, a default configuration data file 124 a isresident within software 124. The predefined rules or configuration datafile direct the operations of the Service Mode. In other cases, if thedefault configuration is not present or is outdated, a new configurationdata file 124 a may be loaded into software 124 or file system 132 a.Advantageously, a configuration data file 124 a permits quick anddynamic control of the Service Mode without substantial changes tosoftware 124.

In one embodiment, the Service Mode operations will request a newconfiguration data file 124 a from a server 120. In other embodiments, aserver 120 will initiate a download of a new configuration data file 124a to the mobile device 118 a-d. The communication of the configurationdata file 124 a may be manually directed or may be scheduled forautomatic operation. In any case, when a new configuration data file 124a is downloaded wirelessly, server 120 and the mobile device 118 a-dform an HTP, UDP, FTP, or other connection to facilitate the download.

In one embodiment, the configuration data file 124 a is very small. Insuch an embodiment, the configuration file 124 a is received as apayload of a single message packet from the server 120. Accordingly, thesmall configuration file 124 a serves many purposes. For example, inenvironments where the data communication services are not robust, asmall data payload increases the likelihood of communication success. Asanother example, the small configuration data file 124 a reduces thepower consumed by the mobile device 118 a-d.

In the preferred embodiment, the collection function of the Service Modegathers a set of Key Performance Indicators (KPI), such as those foundin Table 1. The mobile device 118 a-d is scanned, and one, some, or afull set of all KPI are periodically collected and stored in memory 130a. For example, using an internal timing mechanism of the mobile device118 a-d, the KPI data may, by default, be collected one time every hour,one time every day, or by some other period.

In a preferred embodiment, the KPI data may be collected at the sametime every hour or may be scheduled in another fashion. For example, theService Mode may be configured to scan the mobile device 118 a-d at thesame time each hour within a 24-hour period, and then, at the beginningof the next 24-hour period, the minute within each hour when the KPI arecollected is offset by a predetermined amount. In this manner, forexample, if the predetermined amount of offset is one minute, then aftera 60-day cycle, the mobile device 118 a-d will have been scanned atevery minute at least once throughout a 24-hour time span.

As another example, the KPI data may be collected at any time within aparticular time window. That is, when the mobile device 118 a-d isactive, the current time may be retrieved, and if the current time iswithin a particular time window, then the mobile device 118 a-d isscanned.

After KPI data is collected, the data is communicated in an updatefunction of the Service Mode to server 120. The data may be requested bythe server 120 or the communication of KPI data may be initiated by themobile device 118 a-d. For example, using an internal timing mechanismof the mobile device 118 a-d, which may be the same or a differenttiming mechanism used during KPI data collection, the collected data maybe communicated to the server 120.

In some cases, the Service Mode may have timers set to trigger a KPIdata collection scan of the mobile device 118 a-d at a time whichoverlaps another trigger set to send KPI data to a server 120.Advantageously, the KPI data collection scan is most often allowed tocomplete before the data is updated to server 120. In this way, theupdated KPI data communicated to server 120 is the most recent datacollected on the mobile device 118 a-d.

As previously described, a configuration data file 124 a in memory 130 adirects the Service Mode to perform particular KPI data collectionoperations. The configuration data file 124 a may be updated as neededto change which KPI data are collected, how often the KPI data iscollected, the geographic location of where KPI data are collected, orany combination of these or other parameters.

For example, if it is determined that a particular geographic area inwhich a mobile device 118 a-d operates has sufficient data that wascollected during the morning, and a need for additional data during theevening, then an updated configuration data file 124 a may direct theService Mode to collect KPI data more frequently and only during theevening.

Other factors may also be used to update a configuration file 124 a orto control operations of the Service Mode. For example, since mobiledevices are generally sensitive to power consumption, the type of supplythat is powering the mobile device 118 a-d may also be considered. Inone embodiment, a mobile device 118 a-d that is plugged into ageographically fixed power supply, such as a cradle, will collect allKPI data available on mobile device 118 a-d at a particular rate insteadof just collecting the KPI data directed in a configuration data file124 a at a different rate. Similarly, if the mobile device 118 a-d isplugged into a mobile power supply, such as a car charger, the ServiceMode will operate to collect all KPI data more frequently or at aschedule impacted by geographic location. These and other modes arecontemplated and easily directed by server 120 or by the programmedoperation of the Service Mode.

Storage of the KPI is typically maintained until the KPI are off-loadedto server 120 or another server. In addition, or alternatively, the KPImay also be stored in a cyclical buffer arrangement wherein older datais released so that newer data may be collected.

Communication of the KPI data, the configuration data file 124 a, or anyother communications between the mobile device 118 a-d and server 120are optionally encrypted. By using an available encryption scheme, forexample the exchange of secure keys in a symmetrical, asymmetrical, orother fashion, the information passed between the mobile device 118 a-dand the server are kept confidential. In a preferred embodiment thevalidation of the information operates to prevent fraudulent and/ormalicious information from infecting the mobile device 118 a-d and/orthe server 120. In addition, the validation of the information may alsobe used to govern the amount of KPI data received by the server 120.E.g., legitimate data from a mobile device 118 a-d may be intentionallyrejected if the server 120 determines that the data is not necessary.

FIG. 3C illustrates portions of a non-limiting embodiment of a server120 of FIG. 2. Server 120 is a computing server that includes operativehardware found in a conventional computing server apparatus such as oneor more central processing units (CPU's), volatile and non-volatilememory, serial and parallel input/output (I/O) circuitry compliant withvarious standards and protocols, wired and/or wireless networkingcircuitry (e.g., a communications transceiver).

As known by one skilled in the art, the server 120 has one or morememories 142, each memory comprises any combination of volatile andnon-volatile computer-readable media for reading and writing. Volatilecomputer-readable media includes, for example, random access memory(RAM). Non-volatile computer-readable media includes, for example, readonly memory (ROM), magnetic media such as a hard-disk, an optical diskdrive, a floppy diskette, a flash memory device, a CD-ROM, and/or thelike. In some cases, a particular memory is separated virtually orphysically into separate areas, such as a first memory, a second memory,a third memory, etc. In these cases, it is understood that the differentdivisions of memory may be in different devices or embodied in a singlememory.

Server 120 further includes operative software found in a conventionalcomputing server such as an operating system, software drivers to directoperations through the I/O circuitry, networking circuitry, and otherperipheral component circuitry. In addition, server 120 includesoperative application software such as network software forcommunicating with other computing devices, database software forbuilding and maintaining databases, and task management software fordistributing the communication and/or operational workload amongstvarious CPU's. In some cases, server 120 is a single hardware machinehaving the hardware and software listed herein, and in other cases,server 120 is a networked collection of hardware and software machinesworking together in a server farm to execute the functions of thewireless network performance measurement system 110. The conventionalhardware and software of server 120 is not shown in FIG. 3C forsimplicity.

FIG. 3C illustrates software components loaded into the memory 142 ofserver 120. The memory comprises any combination of volatile andnon-volatile computer-readable media for reading and writing. Volatilecomputer-readable media includes, for example, random access memory(RAM). Non-volatile computer-readable media includes, for example, readonly memory (ROM), magnetic media such as a hard-disk, an optical diskdrive, a floppy diskette, a flash memory device, a CD-ROM, and/or thelike.

A client data daemon 128 b, a data translation module 130 b, a datafiltering module 132 b and a database storage module 134 b areprogrammatic applications that operatively work together to analyze andprocess data recorded in the wireless network performance measurementsystem 110. A database 136 b is used to store raw and processed data.

Database 136 b may be a traditional database, one or more databasesystems, a file system, or any other method known in the art for storingsuch information, or any combination of the above, includingimplementation using distributed computing techniques.

A client configuration manager 138 b configures software running on oneor more mobile devices 118 a-d within the wireless network performancemeasurement system 110. For example, the software running a mobiledevice, 118 a-d is a wireless network user quality data collectionsoftware program. A data control module 140 b processes raw data fromthe mobile devices 118 a-d and distributes the raw data to theprogrammatic applications.

In a preferred embodiment, server 120 sends data to and receives datafrom a large number of mobile devices 118 a-d. Server 120 processes andstores location based mobile device user experience data received fromthe mobile devices 118 a-d. After storage of the processed userexperience data, the wireless network user quality data measurements(e.g., the data) are made available to additional servers, for exampleweb server 126 (FIG. 2), for presentation to consumers. In such anembodiment, the remote computing device (e.g., web server 126) makesrequests for particular data. The server 120 responds to the requests byproviding or otherwise permitting access to the data.

In another preferred embodiment, a unique user location profile may begenerated for one or more mobile devices 118 a-d operating in a wirelessnetwork performance measurement system 110. For example, each mobiledevice identified by server 120 may be given a unique identifier. Theunique identifier may be random number, an Electronic Serial Number(ESN), an International Mobile Equipment Identifier (IMEI), or someother number. In many cases, the number is encrypted. After generatingthe unique identifier, the identifier may be combined with typical,repeated geographic and time information to create the unique userlocation profile.

After a user location profile is generated for a mobile device 118 a-d,the profile is available for use by server 120 to collect particulardata at particular times and in particular locations. In some cases,server 120 can use the profile in a predictive manner, and in othercases, server 120 can use the profile to make decisions about dataretrieval from particular mobile devices 118 a-d.

For example, in order to collect data only in certain locations and asinfrequently as possible, server 120 may actively monitor how manymobile devices 118 a-d are currently in, or predicted to be, in aparticular geographic area or sector during a particular time window.Server 120 may further monitor which sectors have outdated or otherwiseinsufficient data. Server 120 may increase or decrease the frequency ofdata collection of a mobile device 118 a-d (e.g., using a user locationprofile) based on the age, quality, or volume of data previouslygathered. Software in the mobile device 118 a-d is responsive to therequests for data of server 120.

In another example, when the wireless network performance measurementsystem 110 is initialized in a new geographical area, the number of datacollection mobile devices 118 a-d may be small. In such circumstances,the server may collect large amounts of data from each mobile device 118a-d in the entire area. Performance and power considerations areconsidered for each mobile device 118 a-d, so server 120 may onlyrequest data in certain areas of interest, and server 120 may functionto prevent or minimize the collection of data in overlapping areas. Thatis, in areas where there are few mobile device clients 118 a-d, themobile device client may be requested to collect data with greaterfrequency only in areas where server 120 does not currently have data.

Alternatively, server 120 may reduce or suspend requests for data inareas where server 120 does not currently have data. In other cases,server 120 may request data based on one set of parameters, but softwareof mobile device 118 a-d may decide locally whether and how to respondto the request based on a local set of parameters. For exampleparameters that the mobile device 118 a-d may use to decide whether andhow to respond include current battery life, current processorutilization, scheduled events, and the like.

The software in the server 120 may also modify its request for databased on the results of the data.

As examples, if the returned data shows that particular ones of themobile devices 118 a-d are nearly always plugged in to a separate powerand supply and not usually battery powered, the server may send signalsto more frequently use these mobile devices 118 for high power systemchecks, such as GPS sensing, antenna power consumption, or the like.Similarly, if the server sees that some of the mobile devices 118 arerarely used and have a wide available bandwidth in the CPU and also fortransmitting and receiving, it may request additional tests or moreimmediate responses from such mobile devices 118. Alternatively, if thetest results indicate rapid battery depletion, then the server canmodify the commands it sends to reduce the battery usage and conductfewer tests or only low power tests.

In some cases, the wireless network performance measurement system 110is initialized or operating in a geographical area where the number ofdata collection mobile devices 118 a-d is large, e.g., 5000 or moredevices. In such cases, server 120 may further function to adjust thefrequency and/or volume of data collection by a particular mobile device118 a-d. For example, rather than collecting data from every device,server 120 may collect data from only some devices and use techniquessuch as statistical modeling, to create a complete set of wirelessnetwork performance data. As another example, server 120 may direct somemobile devices 118 a-d to collect data of one type and direct othermobile devices 118 a-d to collect data of another type. The wide rangeof data collection directions provided by server 120 in a geographicalarea having many mobile devices 118 a-d reduces the demand for power dueto data collection on each mobile device 118 a-d.

Another technique that may be used by server 120 to direct datacollection only in certain locations and as infrequently as possible isto take advantage of location based services on the mobile device 118a-d. For example, a global positioning system (GPS) is available on somemobile devices 118 a-d. Operating the GPS circuitry, however, uses alarge amount of power on most devices. Accordingly, it is desired tocollect GPS data only when mobile device 118 a-d has changed location bya large distance or is plugged in. For example, the server 120 can senda software command to update the GPS location only after the mobiledevice 118 has been plugged into a power supply for a sufficient time tocharge the battery.

One technique used to minimize the frequency of GPS driven datacollection is to take advantage of the cellular network's capability toprovide the distance of the mobile device 118 a-d from the closestcellular tower or any other cellular tower. The cellular towerinformation, e.g., the distance, is collected by the radio hardware inthe mobile device 118 a-d. This distance data is independently updatedby the radio module on the mobile device 118 a-d even in a power savingmode. By monitoring the distance, the mobile device 118 a-d candetermine when it has moved from a particular position by apredetermined amount, e.g., 30 meters.

Using the distance information from the cellular tower, server 120 candirect whether or not the GPS module of the mobile device 118 a-d needsto be engaged. For example, if the distance to the closest cellulartower has not appreciably changed, then the mobile device 118 a-d maynot be directed to capture new GPS coordinates and new networkperformance data.

The Client Data Daemon 128 b (CDD) of server 120 is the subcomponentresponsible for communicating with mobile devices 118 a-d. A low levelproprietary protocol is used to facilitate communication between the CDD128 b and each mobile device 118 a-d via a communications transceiver.The protocol consists of a predetermined set of commands and responses.The commands and the responses are generally asynchronous packetizedmessages that are communicated between the server 120 and the mobiledevices 118 a-d.

In some embodiments, the server 120, via the CDD 128 b, communicatesconfiguration data to the mobile devices 118 a-d as the payload of asingle packet. Accordingly, the small configuration data serves manypurposes. For example, in environments where the data communicationservices are not robust, a small data payload increases the likelihoodof communication success. As another example, the small configurationdata reduces the power consumed by the mobile devices 118 a-d.

In some embodiments, the commands are grouped together and communicatedbetween the CCD 128 b and the mobile device 118 a-d as a particularconfiguration file. The configuration file may be small or large, buttypically represents rules, or configuration data points, passed to themobile device 118 a-d that are to be followed by the device whencollecting data. In some cases, the configuration file has executablecode that carries out or otherwise directs the rules, but in othercases, the configuration file has only indicators of whichpre-programmed commands that the mobile device 118 a-d will follow.

Within server 120, the proprietary protocol commands and responsesbetween the server 120 and the mobile devices 118 a-d are communicatedthrough the client data daemon 128 b. A retry mechanism of the CDD 128 boperates to provide robust communication of messages between the server120 and the mobile devices 118 a-d. For each transmitted and/or receivedmessage that requires a response, the CDD 128 b tracks whether suchresponse was received and/or transmitted respectively. CDD 128 b furthertracks messages that are transmitted and/or received that do not requirea response. Additionally, the CCD 128 b functions as a load balancerbetween servers 120 so that a large server farm can be created to handlea large number of mobile devices 118 a-d.

When data messages are received from a mobile device 118 a-d, CDD 128 bvalidates, buffers, parses, and partitions the data into predetermineddata points. For example, when a mobile client device 118 a-d transmitsa stream of collected test data, the CDD 128 b will confirm that thestream is valid using a technique such as a CRC check. The CDD 128 bwill further create a set of pointers into the data stream that identifyparticular items of the data such as signal strength, location,connected cell tower, distance from the cell tower, and other data. Andthe CDD 128 b will partition the data into blocks for the DataTranslation Module 130 b.

Communication of the data messages, any configuration data (e.g., aconfiguration file), or any other communications between the mobiledevice 118 a-d and server 120 are optionally encrypted. Availableencryption schemes such as the exchange of secure keys in a symmetrical,asymmetrical, or some other fashion, may all be used. By using anencryption scheme, the information passed between the mobile device 118a-d and server 120 are kept confidential.

In a preferred embodiment the validation of the information operates toprevent fraudulent and/or malicious information from infecting themobile device 118 a-d and/or the server 120. In addition, the validationof the information may also be used to govern the amount of user qualitydata measurements received by the server 120. E.g., legitimate data froma mobile device may be rejected if the server determines that the datais not necessary.

The Data Translation Module 130 (DTM) is a subcomponent of server 120.DTM 130 b is responsible for buffering and converting separate userquality data points from each mobile device 118 a-d into a databasecolumn name. The DTM 130 b provides a mechanism that permits a systemadministrator operating server 120 to change the database 136 bstructure without a need to make corresponding changes on the mobiledevices 118 a-d.

Within database 136 b, one or more of the data points on the mobiledevice 118 a-d have a corresponding column name. In some embodiments,all of the data points have a corresponding column name in the database.DTM 130 b performs the translation of data points to column names via alookup table, a user inquiry, an analysis of a unique identifier of themobile device 118 a-d, or by some other means.

DTM 130 further provides a buffering function for server 120. That is,in some circumstances, a large amount of data is asynchronously receivedfrom one or more mobile devices 118 a-d. In such cases, DTM 130 btemporarily stores some or all of the data until it can be furtherprocessed by other components of server 120.

After data received from a mobile device 118 a-d has been passed throughthe DTM 130 b, the data is analyzed by a data filtering module 132 b(DFM). The DFM 132 b is a subcomponent of server 120 that is operable toremove one or more invalid or uninteresting data points before the datais stored in database 136 b.

Invalid or uninteresting data points occur in situations where a mobiledevice 118 a-d may have streamed only a partial set of data or where themobile device 118 a-d has streamed an entire set of data, but some datafields in the set are not loaded with useful data. Data that is invalid,uninteresting, or otherwise not useful includes empty data points, datapoints from locations that are not interesting to the system, datapoints that are cumulative, and the like.

As described herein with respect to the mobile devices 118 a-d, they maybe in the middle of sending data and the user may request use of thesame resources, such as by making a phone call.

The mobile device 118 a-d would send a hold signal to the server 120indicating that it has temporarily suspended operations for one or morereasons such as a low battery life, user commands, or the like.

In one embodiment, the server 120 will discard the partial data receivedand instruct the mobile device 118 a-d that it has done so. The data canbe resent or discarded, according to the needs of the server. Forexample, if there is a large amount of data already, the server mayelect to discard it; alternatively, if there is a small amount of dataalready, the server may wish it to be resent from the beginning.

In a second embodiment in which transmission from the phone isinterrupted, the server 120 will have the first part of the transmissionand instruct the mobile device 118 a-d that when it is available againto continue sending the data where it left off.

In some cases, the user location profile is used by the DFM 132 b. Forexample, DFM 132 b may use the user location profile to determine thatan abundance of relevant, up-to-date data of a particular type isalready stored in database 136 b. In such circumstances, DFM 132 b mayignore or actively remove some of the data from the mobile device 118a-d.

Further, in some embodiments, DFM 132 b may even update the userlocation profile so that server 120 directs an associated mobile device118 a-d in its future data collection operations. For example, in onescenario, DFM 132 b determines that sufficient data of a particular typeand/or location from a mobile device 118 a-d is already stored indatabase 136 b. In another scenario, DFM 132 b determines that database136 b contains insufficient data of a particular type. In these cases,DFM 132 b may update the user location profile with representativeinformation that server 120 later communicates to the associated mobiledevice 118 a-d.

Upon detection by the DFM 132 b of invalid or uninteresting data points,DFM 132 b operates to remove the invalid or uninteresting data pointsfrom consideration by server 120. For example, DFM 132 b may loadidentifiable indicators such as null pointers into the fields where datawill not be extracted and saved. Alternatively, DFM 132 b may set flagsindicating that particular data fields are not to be saved.

After DFM 132 b identifies data from a data stream that is to be storedin database 136 b, a database storage module 134 b (DSM) is activated.The DSM 134 b is a subcomponent of server 120. DSM 134 b is responsiblefor storing the filtered data points into database 136 b.

Generally, DFM 132 inserts the preprocessed data into database 136 busing a simple and fast insert command, for example a SQL command.Simple commands reduce the load on database 136 b, which permitsdatabase 136 b to support a large number of devices.

In a preferred embodiment, database 136 b, a subcomponent of server 120,is a relational database accessible by Structured Query Language (SQL)commands. Database 136 b maintains and stores the user quality test datareceived from mobile devices 118 a-d, configuration data for mobiledevices 118 a-d, and other data related to the wireless networkperformance measurement system 110. In the preferred embodiment, DSM 134b builds SQL queries to store, retrieve, modify, and delete data indatabase 136 b. DSM 134 further builds SQL queries to perform theadministrative duties necessary to configure and manage database 136 b.

Another subcomponent of server 120 is a client configuration manager 138b (CCM). The CCM 138 b directs high level communications with mobiledevices 118 a-d to manage the configuration and the updating of a clientsoftware application loaded in mobile devices 118 a-d. For example, CCM138 b communicates to mobile devices 118 a-d commands that includedirectives to collect a predetermined set of user quality data points,directives of when to collect data, directives that identify ageographic region from which to collect data, and the like.

CCM 138 is coupled to database 136 b, and CCM 138 b has access to theorganizational structure of database 136 b. CCM 138 b is operable tomanage a large number of mobile device clients 118 a-d by accessing theorganizational structure of database 136 b. That is, CCM 138 is operableto communicate directions to various groups of mobile devices 118 a-d,individual mobile devices 118 a-d, or even to every mobile device 118a-d that is part of the wireless network performance measurement system110.

One function of the CCM 138 b is to perform device management from aconfiguration and software version control perspective. The CCM 138 b isable to instruct one or more devices, which may be divided into groups,to update the current mobile device 118 a-d software to a specifiedversion. In some cases, the CCM 138 b allows server 120 to set up theconfiguration of each device individually, and in other cases, CCM 138 ballows server 120 to configure a group of devices. The CCM 138 b willalso query a mobile device 118 a-d or a group of devices for theirstatus; such as battery strength, flash and memory space available, andother status variables.

In one embodiment, a mobile device 118 a-d will initiate a request to beupdated by server 120. For example, a mobile device 118 a-d will requesta new configuration file from server 120. In other embodiments, server120 will initiate a download of configuration information, such as a newconfiguration file, to the mobile device 118 a-d. The communication ofthe configuration file may be manually directed or may be scheduled forautomatic operation. In such cases, when the new configuration file orother information is downloaded wirelessly, server 120 and the mobiledevice 118 a-d form an HTP, UDP, FTP or other connection to facilitatethe download.

In one embodiment, the configuration file is very small. In such anembodiment, the configuration file is transmitted as a payload of asingle message packet from the server 120. Accordingly, the smallconfiguration file serves many purposes. For example, in environmentswhere the data communication services are not robust, a small datapayload increases the likelihood of communication success. As anotherexample, the small configuration data file reduces the power consumed bythe mobile device 118 a-d.

The server 120 of the wireless network performance measurement system110 presents a structured device management architecture to direct theoperation of as few as one and as many as thousands of mobile devices118 a-d. The structured architecture further permits the data collectionfrom many mobile devices 118 a-d to be synchronized with other mobiledevices.

For example, synchronized data collection permits multiple mobiledevices 118 a-d to be directed with either the same or complementarydata collection techniques. In this manner, the effectiveness of eachmobile device within the wireless network performance measurement system110 may be increased. For example, in some cases, the real time locationof the device (e.g., the user location profile) is used to determinewhich data will be collected. In other cases, different criteria areused to direct the data collection including the type of device, thecurrent application load, the signal strength indication, or the like.

Within the wireless network performance measurement system 110, one ormore mobile devices 118 a-d can be associated with one or moreparticular groups by the CCM 138 b of server 120. The grouping permits aserver 120 to configure multiple mobile devices 118 a-d with a commonset of data collection rules such as a data collection update schedule.

In one example, a plurality of mobile devices 118 a-d is groupedaccording to a geographic area where the plurality of mobile devices 118a-d is operating. The plurality of mobile devices 118 a-d can be furthergrouped into subgroups, and the subgroups may be further divided intomore narrow geographic areas, overlapping geographic areas, or the like.In this way, via multiple CCM 138 b groupings, server 120 cancommunicate customized rules or configuration data to the one or moreparticular groups.

Generally, the mobile device 118 a-d is unaware of which group it iscurrently associated with, but in some cases, during system testing forexample, the mobile device 118 a-d is aware of its group assignment.Instead, most often, each mobile device 118 a-d merely takes directionfrom server 120 in the form of wireless commands. The wireless commandscan direct the mobile devices 118 a-d to perform wireless network userquality data collection operations at different times, at differentlocations, different frequencies, according to the cell tower the mobiledevices 118 a-d are associated with, or the like.

The server 120 is operable to provide direction to mobile devices 118a-d via wireless communication. In one embodiment, communication betweenthe server 120 and the mobile device 118 a-d is facilitated through oneor more data sockets. That is, server 120 opens a data socket to amobile device 118 a-d, which permits the server 120 to direct the mobiledevice 118 a-d when the rules are to be changed, software is to beupdated, or any other control information is to be exchanged. In caseswhere a single socket is opened by the CDD 128, the communication of allcontrol information over the single socket to a single mobile device 118a-d is helpful to efficiently use data network resources.

Server 120 is operable to update and modify the data collection rulesfor mobile devices 118 a-d. In some cases, server 120 configures themobile devices 118 a-d with parameters including a list of data pointsto collect, a list of real-time locations identifying where data will becollected, and an indication of how often data points should becollected. Server 120 may configure each mobile device 118 a-dasynchronously and at any time, or server 120 may configure groups ofmobile devices 118 a-d concurrently.

Several data collection rules determined within the CCM 138 b direct thedata collection operations on the mobile devices 118 a-d. By directingthe mobile devices 118 a-d to operate under a set of changeable rules,the value of the data that is collected by the mobile devices 118 a-dand communicated to the server 120 is improved.

One rule, or configuration datum, passed to mobile device 118 a-d is acell tower list rule. The cell tower list rule includes a list of celltower areas communicated to the mobile device. The cell tower listinforms the mobile device 118 a-d data collection is desired by theserver 120. According to this rule, the mobile device 118 a-d isdirected to monitor the identity of a currently connected cell tower andprovide user quality data to server 120 if the connected tower isidentified on the cell tower list. Subsequently, if the mobile device118 a-d connects to a new cell tower, the mobile device 118 a-d isdirected to re-scan the cell tower list and determine if data collectionhas been directed. In some cases, server 120 directs the mobile device118 a-d to disable the cell tower list, which is a direction to themobile device 118 a-d to collect data in all areas.

The cell tower rule may further take advantage of the user locationprofile generated for a mobile device 118 a-d. For example, the knownlocation of the cell tower may cooperatively be used with the presentlocation of the mobile device 118 a-d and the user location profile tocollect particular data at particular times and in particular locations.Such information, when combined with a particular cell tower, providesan opportunity to collect very specific data related to the operation ofthe wireless network 122.

Two other rules communicated by server 120 to mobile devices 118 a-d area data speed test frequency rule and a data latency test frequency rule.The rules include a value of how many seconds the mobile device 118 a-dwill wait between each data speed test or each data latency testrespectively. Server 120 can direct mobile device 118 a-d to run a dataspeed or data latency test at a single frequency, at a particularpattern of frequencies, a random frequency, or at some other interval.Further, individual cell towers or groups of cell towers in the celltower list rule (e.g., the user location profile) can correlateindependent values for the data speed and data latency test frequencyrules. In addition, when the cell tower list is disabled, independentdefault values may be specified for the data speed and data latency testfrequency rules.

Through the data sockets, a particular command channel can be opened upfrom server 120 to a mobile device 118 a-d. The command channel may beinitialized by either the server 120 or the mobile device 118 a-d, andeither unit may initiate a control cycle. The command channel permitsthe communication of control information between the server 120 and themobile device 118 a-d.

Using the command channel between the server 120 and a mobile device 118a-d, server 120 can interrogate the mobile device 118 a-d for thecurrent client application software version. For example, server 120 maysend a “VERSION” command to a mobile device 118 a-d. The mobile device118 a-d will return the current client application software versionnumber to server 120. In one embodiment, the version number will be asingle floating point number.

Using the command channel between the server 120 and a mobile device 118a-d, server 120 can update the client application software of the mobiledevice 118 a-d to different version. Generally, after the clientapplication software of the mobile device 118 a-d has been updated, thenew version of client software will restart without the need to rebootthe mobile device 118 a-d.

When the server 120 updates the client application software of themobile device 118 a-d, the update may be small or large. For example, insome cases, server 120 downloads a small configuration file part of theclient application software, and in other cases, server 120 completelyreplaces all of the present client application software on the mobiledevice 118 a-d with a fresh version. In some cases, the clientapplication software is encrypted, encoded, and/or otherwise compressedfor reasons that include security, privacy, data transfer speed, datacost, or the like.

The command channel between the server 120 and the mobile device 118 a-dcan be used for many other commands. For example, an UPDATE command isuseful for sending a name string from the server 120 to the mobiledevice 118 a-d. A TIME_SYNC command is useful for synchronizing thecurrent time, date, day or other chronological information betweenserver 120 and one ore more mobile devices 118 a-d.

The use of particular rules for data collection, along with the commandchannel for communicating commands, permits the server 120 to direct amobile device 118 a-d to collect significant data related to thewireless network performance measurement system 110. For example, in apreferred embodiment, Table 1 lists several data sets and individualdata points, which server 120 may direct a mobile device 118 a-d tocollect.

A non-limiting example of the operation of server 120 is now described.In the example, a particular set of rules (e.g., a configuration file)are defined by CCM 138 b in order to direct a group of mobile devices118 a-d. The group information is retrieved by the CCM 138 b via queriesto database 136 b. CCM 138 b, after building the commands and datastructures to direct the mobile device 118 a-d group, communicates thedirectives to the group of mobile devices 118 a-d via the CDD 128 b.

When mobile devices 118 a-d collect information as directed by server120, the mobile devices 118 a-d asynchronously pass the information toserver 120 via CDD 128 b. The data is collected through the DTM 130 b,filtered by the DFM 132 b, and stored in database 136 b via DSM 134 b.After the data has been stored in database 136 b, DCM 140 analyzes andprocesses the data. DCM 140 b further communicates a representation ofsome or all of the data to a web server for presentation to users.

FIG. 4 illustrates one embodiment of dialog boxes presented on a website150 as described herein. A first dialog box 152 permits a user to enterdetails of one or more particular geographic locations. A second dialogbox 154 permits a user to enter one or more wireless cellular networkprovider indicators. A third dialog box 156 permits a user to identify aparticular set of KPIs that represent data to be displayed on website150.

In one embodiment, the user may be permitted to provide input to theserver 120 for data to collect. Normally, the server 120 collectsinformation from the mobile devices 118 a-d on a master program. Aviewer of the web portal data may wish to have additional or moredetailed information in some areas of particular types. For example, auser may wish to have additional power supply information from one areaor additional signal coverage information or a more complete list ofdropped calls. In these instances, the web viewer will input to theircomputer a request to increase the collection rate of one or more KPI inan area. This request is input to server 120, which will follow theinstructions and increase or decrease such data collection from theindividual mobile device 118 a-d. Of course, the request for more datafrom selected areas needs to be coordinated with the number of devicesin that area and the needs of the mobile phone users in that area. If aweb viewer inputs a request to increase a high power or data intensiveKPI, such as GPS tracking or one using high CPU resources, the localmobile devices 118 a-d may only partially comply in order to keep theirusers from being inconvenienced. The web user may be given anotification that his request is likely to be burdensome to the usersand that only some of his request will be fulfilled.

After the geographic area, wireless cellular network carrier, and KPIinformation has been identified by a user, a representative map of theidentified geographic area is brought to the foreground of website 150.Overlayed onto the geographic area, a set of coded grids representing arange of KPI averages for the particular area are displayed. Forexample, color coded grids representing signal strength.

In other embodiments, different KPI information, for example, cell toweridentifiers (e.g., ID, name, type, etc.) is also displayed on a map. Insuch an embodiment, the screen data output may overlay one or more celltowers proximate to the particular geographic area or location. Further,the output may show an indication of network coverage of the cell towersderived from performance quality measurements of one or more mobiledevices previously operated proximate to the identified geographiclocation.

FIG. 5A illustrates an embodiment of a selected geographical mappresented on the website 150. FIG. 5B represents a set of coded gridsoverlying the geographical map of FIG. 5A. The coded maps are shown asvariously patterned grids in FIG. 5B for simplicity, however, it isunderstood that the grids may be colored, patterned, and/or otherwiseseparately identifiable in any like manner.

In a preferred embodiment, the maps and grids illustrated in FIG. 5A ofwebsite 150 are interactively controllable by a user. For example, as auser pans in and out of any particular area with a pan control embeddedin software 142, surrounding grids would either be added to orsubtracted from the KPI averaging calculations and presentationaccording to the user's operation of the pan control. As anotherexample, a user could use a computer mouse to drag a displayed pointerover a grid where more granular details of a particular geographicalarea are desired. By selecting the particular grid, website 150 woulddisplay additional details about the particular KPI and/or thegeographical location. In one embodiment, selecting the particular gridin order to request more information causes website 150 to present atiled overlay pop up window having additional individual details thatmake up the KPI averaging color code for that specific grid area.

An embodiment of website 150 facilitates the comparison of wirelessnetworking services offered by several network service providers. Thatis, KPIs of one network service provider may be conveniently comparedagainst the same KPIs of one or more other network service providers. Bycomparing KPIs of several network service providers for the samegeographic area, a user may learn which network service providerprovides the most desirable level of service as measured by the KPI forthe selected area.

In order to compare one network service provider against another networkservice provider for a given KPI, a toggle control on the website 150may be used to switch between coded grids. That is, for easy and fastcomparison, a user could use one or more mouse key clicks to alternatelyor concurrently display the coded grids.

In another example of comparing network service providers, wirelessnetwork performance software 142 may be functionally coded toautomatically compare KPIs of one or more network service providers in aparticular area. That is, a control dialog box may be presented thatgives the user an opportunity to automatically find a “best supplier”for the selected KPI in the selected region. If the user operates thecontrol box in order to cause the automatic operation, then software 142would automatically mine a database of server 120. The results of themining operation represent the “best supplier,” and the results arepresented on the website 150. The automatic functionality, which forexample selects the best supplier for a given KPI with grid details fora given region, includes a Best Carrier Wizard as described herein.

Users of website 150 are provided with an opportunity to make commentson the service offered in a particular location by a particular carrier.In one embodiment, the user's comments are tagged to the particular KPIand map of a geographical area. Subsequently, other users may view thecomments, and take an opportunity to provide additional comments.

In some cases, the users of website 150 are also users of mobile devices118 a-d that collect the data which populates website 150. In othercases, users of website 150 are researching KPI information for aparticular location with a particular mobile device. In still othercases, the user is merely seeking information to help understand aparticular user experience that the user has experienced. In cases wherethe user does not operate a data collecting mobile device 118 a-d,website 150 offers the user an opportunity to operate their mobiledevice as a data collecting mobile device 118 a-d.

More particularly, website 150 provides a portal for users to contributeKPI content. That is, the user may “sign up” to the website 150 anddownload the KPI client application to their mobile device. In somecases, website 150 provides one or more portals to download a clientapplication targeted to operate on a particular type of mobile device.In some cases, website 150 further embodies incentive programs such ascontests, merchandise earning credits, frequent contributor awards, andthe like to encourage users of mobile devices to download the KPI datacollecting client application onto their mobile device.

For example, one or more contest programs assign points to areas onwebsite 150 such as particular geographic coverage maps. Under such anincentive program, geographic areas which typically have less mobilecommunication traffic and are thereby more difficult areas in which tocollect KPI metrics, may accrue more points or may give more weight toaccrued points than areas with heavy user mobile communication traffic.

When users collect KPI data from the geographical points of interest,the user's ID will be assigned points designated for the given area.Points are collected by users throughout the contest period. In somecases, a user having the most points collected at the end of the contestis given a reward stipend for winning the contest. In other cases, usersreceive different reward stipends proportional to the number or weightof the points they have collected. In still other cases, users collectpoints on an ongoing base. Once a user collects a given amount ofpoints, the points can be redeemed for money, prizes, merchandise, orother incentive rewards.

The website may permit a user to customize the KPI's that are collectedfor his viewing from his own mobile device. In some embodiments, if theuser of the mobile device is the one directing the KPI collection, thenormal power roving and CPU consumption guards can be modified ordropped so that the user's phone may in fact have degraded performanceor a shorter battery life because of the tests. But, because the userhimself is deciding to perform the KPI collection, this may be permittedto a greater degree than would otherwise be allowed.

One or more “Select Best Carrier” (SBC) wizards are embodied in wirelessnetwork performance software 142 and presented on website 150. An SBCwizard allows a user to determine which wireless network carrier has the“best” KPI average for a particular facet of the network coverage, for aparticular mobile device, and for a particular geographic area. In somecases, several KPI averages and/or several geographic regions may beaveraged during the automatic selection of a “best” carrier. The “best”carrier may be the one with the strongest signal strength, greatestsignal coverage area, fewest dropped calls, or some other calculablemeasurement.

In one embodiment, users of website 150 will operate a wizard by drawingone or more geographic boxes around particular areas of a geographicalcoverage map. In another embodiment, the user may select pin-points on acoverage map where the user intends to use a particular mobile device.Subsequently, after responding to one or more profiling questions suchas ranking feature importance, the SBC wizard programmaticallydetermines which network carrier will provide satisfactory service. Insome cases, the SBC wizard ranks the particular offerings of severalcarriers and presents the results of the rankings to the user.

KPI Information supported on website 150 as described herein may also becollected from website 150 and reformatted to support other third partywebsites. For example, in one embodiment, another website that supportscontent complementary in nature to the content of website 150 would havethe ability to serve some of the detailed KPI content of website 150 onthe other website.

An additional, optional feature of website 150 is to provide a single,centralized website where users have access to a large amount ofinformation related to wireless network coverage and mobile devices. Theinformation content includes information proprietary to website 150,information shared amongst multiple sources, links to other informationsources and the like. For example, the information content may includeproduct reviews, best practices/white papers, newsletters, user feedbackforums, webcasts, an online store, an alarms section, and the like.

A “Product Reviews” section of website 150 includes a wide range ofinformation content ranging from an analysis of the details of specificmobile devices to comparisons of different types of cell-phone plansoffered by one or more communication network carriers. The productreviews may also include details and user ratings of the customerservice provided by the communication network carriers. In some cases,the information can broadly cover all geographic areas, and in othercases, the information can be representative of a particular geographiclocation.

A “Best Practices” section of website 150 includes information producedto help a user manage their mobile device platform more effectively. Forexample, some information may describe how to set up email. Otherinformation may describe how to manage power usage. Still otherinformation may describe how to address cellular service billingdiscrepancies and errors.

A “Newsletter” section of website 150 includes information supportingspecific areas related to mobile communications. Users of website 150may research or sign up to receive information related to particularareas of interest. For example, users may sign up to receive periodicinformation related to new releases of mobile operating systems. Asanother example, users may research information related to new mobiledevice product offerings from particular device manufacturers. As stillanother example, users may receive information related to particularnetwork service providers or to the entire cellular wirelesscommunication frequency regulations.

A “User Feedback/Forums” section of website 150 includes informationand/or comments posted by other users of website 150. In some cases, theinformation and/or comments may be organized by topic, by date, or bysome other means to answer questions or to provoke discussion. Users mayalso request help from other users of the forum to resolve a particularissue. In such cases, information and/or comments may be organized asparticular topical threads having a series of comments and replies.

A “Webcasts” section of website 150 includes information related toparticular mobile device usage or other areas of mobile communication.For example a collection of video clips include how particular mobiledevices are set up right out of the box, how particular mobile devicesare upgraded, configured with data collection software so as to operateas a data collection mobile device 118 a-d, how particular devices areused for text messaging, and the like.

An “Online Store” section of website 150 includes direct to consumersales of mobile devices, mobile device accessories, mobile communicationservice plans, and other goods and services. The online store may alsoinclude links to third party providers of a wide range of same ordifferent goods and services.

Advertising content of website 150 may take many forms. For example,advertising could populate each page of the website 150 including theKPI look up pages and the various centralized website pages. Advertisingcould be location-based to target the geographical location of areasselected by the user. Advertising could further be content-based totarget the type of information related to the navigation trails of theuser through the information content of website 150.

As described herein, the wireless network performance softwareapplication 142 may serve a variety of purposes. For example, wirelessnetwork performance software application 142 may host a website 150,perform averaging calculations, generate additional network performancedetails related to a location of particular focus, implement “best”wizards for a variety of purposes, mine server 150 for additional data,and many other things.

One example of an averaging calculation used by many features of thewireless network performance software application 142 is a compositescore (CS) that can be used to comparatively measure wireless networkquality in a variety of ways and amongst a variety of elements. The CSis representative of wireless networking performance quality of aparticular mobile device and wireless network user experience, and theCS may be relative to some or all of the available key performanceindicators. For example, via the CS, the network quality can be comparedbetween a plurality of wireless network service providers. In anotherexample, the CS can be used to compare network quality over varyinggeographic parameters. In still another example, the CS is useful tocompare wireless network quality over time. Clearly, as will be evident,the CS is useful for many things.

In a preferred embodiment, three composite indicators are combined in anaveraging algorithm to form the CS, although more or fewer indicatorscould be used. A set of geographically based wireless networkingperformance data is parsed according to the chosen key performanceindicators, and the data is used to develop the CS. Also in a preferredembodiment, the CS is created for the smallest measurable geographicarea, and multiple CS measurements are mathematically combined to form acomposite CS for a larger geographic area of interest.

With respect to the composite indicators, a wide range of measurementsand statistical calculations may be made on raw and/or derived data. Forexample, in some cases, the CS is calculated with a mathematical average(a mean) of measurements taken and recorded by one or more mobiledevices. In other cases, different statistical calculations, as known byone skilled in the art, may be used to form the composite indicators.For example, raw or derived data may be created in part or according toa median, a mode, a weighted average, a standard deviation, a variance,and the like.

In the preferred embodiment, a first composite indicator used tocalculate the CS is a measure of signal quality in the wireless network122. Signal quality may be signal strength, signal to noise ratio, oranother signal integrity value. The first composite indicator of oneembodiment, also called a Signal Quality Index, is illustrated inEquation 1.

SignalQualityIndex=A*(GridSQ)  (1)

Where:

-   -   A is a Signal Quality Weighting Factor;    -   GridSQ is a measure of signal quality in a single, smallest        geographic area.

The measure of signal quality, GridSQ, in the Signal Quality Indexcalculation is relevant to a selectable geographical area. For example,in the geographical map of FIGS. 5 a and 5 b, an average signal qualitymay be a mathematical average of measurements made within a single gridor a mathematical average of measurements made over a collection ofgrids. In this way, signal quality information about a selectablegeographical area is chosen. In the preferred embodiment, the measure ofsignal quality is made for each grid in the area of interest, and a CSis calculated for each grid.

Upon selection of the geographic area, a Signal Quality Weighting Factoris applied to the measure of signal quality. The signal qualityweighting factor in a preferred embodiment is determined based on, forexample, what a particular user of the CS values. For example, if a uservalues voice quality, then a particular signal quality weighting factormay be chosen to favor a desirable network signal quality that performsvoice communications well. In another example, if a user values ShortMessaging Service (SMS) quality, then a different signal qualityweighting factor may be chosen to favor a network having a desirable SMSmessaging integrity. Accordingly, many factors may be used to determinethe signal quality weighting factor.

In the preferred embodiment, a second composite indicator used tocalculate the CS is a measure of data quality in the wireless network122. The second composite indicator, also called a Data Quality Index,may be a weighted determination of how much non-voice traffic can becommunicated through the wireless network 122 to a particular mobiledevice. In some cases, the particular geography, the type of networktechnology, the time of day, the size of packets, and many other factorsimpact how much data can be communicated through the network. The secondcomposite indicator of one embodiment, the Data Quality Index, isillustrated in Equation 2.

DataQualityIndex=B*(TTPoints/TotalPoints)+C*(AvgDS/MaxDS)

Where:

-   -   B is a Data Type Weighting Factor;    -   TTPoints, or Technology Type Points, is a count of data        measurements taken for each of the several Technology Types of        cellular technologies in the total area of interest;    -   TotalPoints is a count of data measurements taken for all of the        several types of cellular technologies in the total area of        interest;    -   C is a Data Speed Weighting Factor;    -   AvgDS, or Average Data Speed, is a calculated average data        transfer speed based on measurements taken in the total area of        interest;    -   MaxDS, or Maximum Data Speed, is a calculated average of a        maximum data speed for the total area of interest.

The Technology Type Points in the Data Quality Index calculation isrelevant to a total selectable geographical area. For example, in thegeographical map of FIGS. 5 a and 5 b, a particular grid represents asmallest measurable geographic area. As empirically determined, thecellular communication characteristics of one grid may provideinformation to predict the characteristics of neighboring grids (i.e.,larger geographic areas). For example, it has been determined that acellular network operating with third generation technology (3G)transfers data faster than an area operating with second generationcellular technology (2G). It has also been determined that in an areaoperating with both 3G and 2G technology, the data speed may be verygreatly impacted in ways that are not generally predictable or expected.Accordingly, in one embodiment, the TTPoints measurement is a differencebetween all data speed measurements taken in a 3G network and a 2Gnetwork taken in ratio to all of the data points measured in the totalarea of interest.

The TotalPoints element in the Data Quality Index calculation is a countof data measurements taken for all of the several types of cellulartechnologies in the total area of interest. For example, if two milliondata tests are run in a particular area on a 3G network and threemillion data tests are run in the same particular area on a 2G network,the TotalPoints element would be five million.

In the first part of the Data Quality Index calculation, it isobservable that the ratio between data points of one type of cellulartechnology and other types of cellular technology are calculated.

Upon calculation of the data type, or technology type, ratio, a datatype weighting factor is applied to the ratio. The data type weightingfactor in a preferred embodiment is determined, as described above,based on the number and type of different cellular data technologiesoperating in the selected total area of interest. For example, in somenetworks, it has been observed that data transfer operations in a 2Gnetwork execute more rapidly than in a mixed 2G and 3G network. Similarempirical observations can be made with a wide variety of cellulartechnologies and combinations. Accordingly, the data type weightingindex can be chosen to account for a wide variety of conditions.

The Average Data Speed in the Data Quality Index calculation in apreferred embodiment represents the average of all data transfers in thearea of interest without respect to technology type. In someembodiments, thousands and millions of data speed tests are executed inthe smallest measurable geographic areas. In such embodiments, theaverage data transfer speed for the smallest measurable geographic areais combined and averaged with all of the other data speed averages (persmallest measurable geographic area) in the total area of interest.

The Maximum Data Speed element in the Data Quality Index calculation isa global, statistically determined, maximum data speed for the totalarea of interest. For example, as discussed with respect to differentcellular technologies, some configurations of networks and devicesperform better than others. By empirically measuring or predictablycalculating, a maximum expected data speed in the total area of interestis determined. The ratio of average data speed and maximum data speed isthen calculated.

A Data Speed Weighting Factor is then combined with the average dataspeed/maximum data speed ratio. The data speed weighting factor isuseful for adjusting the relative weight of the data speed ratio in theCS calculation. For example, emerging technologies that may be intesting modes, actual measurements that have been skewed for knownreasons, packet sizes, and the particular intrinsic value placed on datatraffic speed within the CS calculation are each potential factors inthe choice of data speed weighting factor.

A third composite indicator used in a preferred embodiment to calculatethe CS is a measure of network reliability in the wireless network 122.The third composite indicator, the Network Reliability Index, may be aweighted determination of the likelihood of encountering errors on thewireless network 122. For example, how often a particular mobile deviceshows “no bars” to indicate an unavailable network is generallyimportant to a determination of network quality. Additionally, thenumber of errors encountered when transferring data packets, the numberof digitized voice packets that are dropped and account for poor voicequality, and the number of times a relatively stationary mobile devicenevertheless switches between servicing cell towers are all factors usedto determine network reliability. A third composite indicator, theNetwork Reliability Index, of one preferred embodiment of a CScalculation is illustrated in Equation 3.

NetworkReliabilityIndex=D*(R1/NDC)+E*(R2/ZS)+F*(R3/SS)  (3)

Where:

-   -   D is a No-Data Connect Weighting Factor;    -   R1 is a global average No Data Connect Error Rate;    -   NDC is a No Data Connect Error Rate in the total area of        interest;    -   E is a Zero-Signal Weighting Factor;    -   R2 is a global average Zero Signal Error Rate;    -   ZS is a Zero Signal Error Rate in the total area of interest    -   F is a Swing-Signal Weighting Factor;    -   R3 is a global average Swing-Signal Error Rate;    -   SS is a Swing-Signal Error Rate in the total area of interest.

The global average No-Data-Connect Error Rate, R1, in the NetworkReliability Index calculation is relevant to a total selectablegeographical area. Generally, a no-data-connect condition occurs when amobile device can associate with a particular network, but the mobiledevice is unable to connect to services sufficiently robust to actuallytransfer data. In a preferred embodiment, R1 is a “national average” ofan expected number of no-data-connect errors for a given geographicarea; however a different average could be used in other cases. Forexample, a time component over which to measure errors could beconsidered, a particular technology type could be considered, a type ofgeography could be considered, and many others.

In ratio with the global No-Data-Connect Error Rate, a No-Data-ConnectError rate, NDC, measured in the total area of interest is considered.In the preferred embodiment, the ratio of measured no-data-connecterrors NDC to the global average R1 is weighted with a No-Data-ConnectWeighting Factor D.

In a second element of the Network Reliability Index, a global averageZero-Signal-Error Rate, R2, in the Network Reliability Index calculationis processed. The Zero-Signal-Error Rate, R2, is relevant to a totalselectable geographical area. Generally, a zero-signal-error conditionoccurs when a mobile device cannot even associate with a particularnetwork. In some cases, the mobile device cannot even detect thepresence of a wireless network. In some embodiments, R2 is a “nationalaverage” of an expected number of zero-signal-error conditions for agiven geographic area, but different averages could also be used. Forexample, a time component over which to measure errors could beconsidered, a threshold of signal detectability could be considered,known anomalies in geography or network characteristics could be takeninto account, or the like.

In ratio with the global Zero-Signal-Error Rate, R2, a Zero-Signal-ErrorRate, ZS, measured in the total area of interest is considered. In thepreferred embodiment, the ratio of measured zero-signal-errors ZS to theglobal average R2 is weighted with a Zero Signal Weighting Factor E.

A third element of the Network Reliability Index in the preferredembodiment is also processed. A global average Swing Signal Error Rate,R3, in the Network Reliability Index calculation is relevant to a totalselectable geographical area. Generally, the swing signal condition is afactor accounting for a particular mobile device quickly associatingbetween two or more cell towers. This condition can occur for manyreasons, including as a result of geography or atmospheric conditions.In other cases, the swing condition occurs as a cell tower begins toreach saturation in manageable traffic and so adjusts its power output.In still other cases, a swing condition occurs as a result of physicalmotion of the mobile device. In some embodiments, R3 is a “nationalaverage” of an expected number of swing signal conditions for a givengeographic area, but different averages could also be used. For example,a time component over which to measure errors could be considered, adensity threshold of population or traffic could be considered, or othernetwork factors could be taken into account.

In ratio with the global Swing Signal Error Rate, R3, a Swing SignalError Rate, SS, measured in the total area of interest is considered. Inthe preferred embodiment, the ratio of measured swing signal errors SSto the global average R3 is weighted with a Swing Signal WeightingFactor F.

The weighting factors used in the Network Reliability Index calculation,D, E, F, can be adjusted or predetermined for a wide variety of factors.In some cases, the factors are based on measured, empirical data, and inother cases, the adjustments can be made based on network qualitycharacteristics prioritized by a user.

In a preferred embodiment, the CS is calculated as a summation of thethree composite indexes discussed herein. A preferred embodiment CS isillustrated in Equation 4.

CS=SignalQualityIndex+DataQualityIndex+NetworkReliabilityIndex  (4)

Other factors could be used to calculate the CS for a small or largegeographic area, and/or the composite indexes could be combined in manyways. For example, in some cases, the CS is normalized to always producea result in the range of 1 to 10. In additional cases, the CS isformatted for output to a display or other computing device.

By normalizing CS calculations, a user of the network performancesoftware application 142 can very easily compare network quality betweenvarious network service providers and over various large and smallgeographical locations. In addition, the user can compare the differentvoice and data services provided by the various network providers usingCS calculations produced with an even-handed, un-biased testmethodology.

Another example of a calculation used by certain features of thewireless network performance software application 142 is a predictivecomposite score (pCS). A pCS value is a forecast of expected performancequality of a wireless communication network. A pCS can be calculated fora smallest measurable geographic area or a geographic area of anotherchosen size. Multiple pCS measurements can be mathematically combined toform a composite pCS for a larger geographic area of interest. A pCS canbe used to provide an indication of wireless network quality that can beexpected in an identified geographic area.

In the embodiments discussed herein, different parameters can becombined in various ways to predict different components of wirelessnetworking quality. For example, a first pCS value associated with acertain geographic area predicts the confidence with which a wirelesscall can be successfully connected. A second pCS value associated withthe certain geographic area can predict the confidence with which awireless call will remain connected, including, for example, through oneor more handoffs. A third pCS value may predict the confidence withwhich a certain data rate can be achieved, maintained, or even whetherdata can be passed at all. Additional pCS values can also be calculated.Furthermore, similarly calculated pCS values for different networkservice providers can be used to compare the predicted network qualityof one service provider to other service providers.

Upon calculation of pCS values, predicted wireless networkingperformance quality can be compared between particular networks,particular mobile devices (e.g., between two or more different models ofmobile devices), particular geographic locations, particular seasons,times of day, and the like. For example, via the pCS, network qualitycan be compared between pluralities of wireless network serviceproviders during different times of the day. In another example, the pCScan be used to compare network quality over varying geographicparameters—even where only a small amount of data or no data at all haspreviously been collected. Over time, a pCS value can be calculateddynamically to better understand how the expected quality of a networkwill evolve over time.

In an exemplary embodiment, spatially aggregated data from one or moregiven geographical locations is geostatistically combined to form thedesired pCS. Speaking broadly, geostatistical calculations include theapplication of statistical analysis to any data that has a geographicalor geological component. A statistical analysis includes the applicationof mathematical formulae to the collection and interpretation ofquantitative data and the use of probability theory to estimate unknownparameters and/or data members. Geostatistics therefore includes thestatistical analysis of data associated with features of the earth'ssurface such as topography, terrain, climate, vegetation or lackthereof, building density, and the like. Geostatistical calculations, asused herein, include parameters associated with geographic features, andin some cases, the geostatistical calculations also include parametersassociated with other features such as population density,electromagnetic spectrum, network technology, and the like.

Geostatistical calculations can include kriging algorithms,spatio-temporal averaging, data mining, modeling, and many othertechniques. In some embodiments, certain techniques are used to createone type of pCS, and in other embodiments, different techniques are usedto create another type of pCS. For example, in areas of heavy populationdensity, spatio-temporal averaging techniques may influence pCScalculations more than other techniques. In areas where actual data islimited or aged, kriging techniques may be relied on more heavily toproduce a pCS. In still other areas, where an aperiodic event is takingplace (e.g., severe storm, a major sporting event, etc.), data miningand modeling techniques may be used to produce a pCS.

In some embodiments discussed herein, kriging algorithms perform spatialinterpolation by applying weighting factors based on the distancebetween geographic points where key performance indicator data has beencollected. The kriging techniques model the correlation structure of thenetwork data as a function of distance and may include information aboutany known covariance between the predicted quantity and other spatialinformation (co-kriging).

As applied to embodiments described herein, kriging algorithms provide aprediction of a quantitative measure (e.g., key performance indicators).If a particular mapped grid includes little or no actual measured data,or if the data that contributes to the grid was collected at adetermined time too far in the past (i.e., the data is “stale”), thennetwork performance predictions can be made about the likelihood, forexample, of call completion, call quality, data throughput, and the likein and around the grid. The predictions, which are derived fromspatio-temporal models of the correlation structure of training data(e.g., kriging or co-kriging in the case of purely spatialinterpolation), provide a consistent, quantifiable, process for datainterpolation. Stated differently, the interpolation algorithms bringanalysis based estimation to composite score calculations, thus enablingthe predictive composite score (pCS).

FIG. 5C illustrates a geographical area having a set of overlying codedgrids. In some embodiments, the coded grid segments of FIG. 5C areconstituted similar to the grid segments of FIG. 5B (e.g., color coding,textured, interactive, etc.) wherein the coding of the individual gridsegments provides information related to the performance quality of awireless communication network. Individual grid segments in FIG. 5C canbe coded to represent key performance indicator (KPI) data of theunderlying area including KPI data that is predictively weighted.

FIG. 5C includes an arrangement of “dots.” The dots represent geographicpoints where actual network performance data has been collected. The“dots” of FIG. 5C may optionally be visible on the display of thegeographic area, or alternatively, the dots may not be displayed. Thetime, quantity, quality, and geographic location where data has beencollected is used to produce the coding of the grid segments. In somecases, the data collected in a particular area of an individual gridsegment is used to independently generate the displayable coding for therespective grid segment. In other cases, the data collected from onegrid segment contributes to coding of other adjacent or nearby gridsegments.

In FIG. 5C, marked axes are illustrated. The marked axes, which may ormay not be used, uniquely identify individual grid segments. In somecases, the unique identification of the grid segments persists only withrespect to the current display. In other cases, the entire earth ismapped and each grid segment is universally unique. In geographicinformation systems, a spatial reference scheme provides a coordinatereference or other codified name to clearly identify, in two andoptionally three dimensions, any point on the target object (e.g.,earth). In some cases, the identifiers in the spatial reference schemeare displayed, and in other cases they are not. Typically, theidentifiers are used in the kriging procedures to generate a predictivecomposite score (pCS).

As illustrated for simplicity in FIG. 5C, a sprinkling of data sampleshas been collected. Several reasonable predictions can be formed. Forexample, a prediction as to the “consistency” of network quality in anindividual grid segment can be formed in grid cells that have multiplesamples. Grid segment 8-E is chosen for the sake of explanation. Withingrid segment 8-E, a data sample is collected at one point near aboundary of the grid segment. When one or fewer points are collectedwithin a grid segment, a method called “block kriging” is used to createan estimate over the entire block. Block kriging is similar to simple(point) kriging in that the estimate is formed by considering thecorrelation between sampled points and many or all points within thegrid segment.

Making a prediction about the reliability of one or more determinednetwork quality parameters is also reasonable. For example, it isreasonable to predict that the determined network quality parameters indata segment 8-E will be less certain than the parameters of datasegments 8-D and 8F, which each have more than one collected datasample.

It is further reasonable to predict that the determined network qualityparameters in grid segment 8-E will have similar values to correspondingparameters in segments 7-E, 8-D, 9-E, and 8-F, which share adjacent gridsegment borders. Additionally, the determined network quality parametersin grid segment 8-E may also be reasonably predicted to have close orsimilar values to corresponding parameters in nearby grid segments 7-D,9-D, 9-F, and 7-F. Knowledge about the data associated with gridsegments that are further away may “influence” a prediction about thedata parameter values of a particular grid segment (and vice versa).

Generally speaking, it is recognized that data collected at a variety ofparticular geographic locations can be related to an expected networkperformance quality at nearby locations. The influence of any one sampleor datum (or any amalgamation of samples or data) on a prediction ofnetwork performance quality can generally be strong or weak based on avariety of factors, such as those listed in Table 2.

TABLE 2 Factors that influence network quality in a geographic area.Distance; Degradation of Carrier Signal Over Terrain Tower Locations;Including Distance to a Mobile Device and Distance Between Towers (moreoverlap raises a risk of dropping a call during a handoff) TowerCapacity Signal Integrity/Quality; Including Signal Strength and Signalto Noise Ratio (SNR or EC/IO) (as may also be analyzed against recordedcall data (crowd sourcing)) Terrain Attributes; Including Foliage,Topology, etc Percentage of Building Coverage Population Density Time ofDay/Time of Year Weather; Including Terrestrial and ExtraterrestrialEvents Traffic Patterns; Including Network Traffic and Movement ofDevices Through a Given Geographical Area Carrier/ProviderInfrastructure; Network Technology Frequency of Wireless Spectrum in UseOther Factors

Based on factors such as those listed in Table 2, determinations can bemade to include or exclude data collected in certain grid segments. Ifsuch data is included, additional determinations can be made to increaseor decrease the importance of the data collected from surrounding gridsegments. In short, it is reasonable to predict that data collectedclose to a certain grid segment should be highly related to that certaingrid segment, and data collected further away will be less related. Datacollected too far from the certain segment by distance or by some othercharacteristic should have no influence on the calculated networkperformance parameters of the certain segment. Accordingly, a range ofinfluence of some data on other nearby data is recognized and applied incertain embodiments. Herein, the range of influence can also beexpressed as an estimator or weighting factor.

In some embodiments, the range of influence is standardized, which helpsto avoid over or under estimating an individual grid segment or group ofgrid segments. One mechanism to standardize to range of influence is toset the sum of weights to one. By this setting, the network qualityparameter is treated consistent, as a whole, throughout the gridsegment. It is recognized that within any given segment, a particularparameter may be anomalous at a particular location (e.g., making atelephone call from inside a parking garage or from the roof of a highrise building located in the grid segment), but overall, a consistencyof the parameter throughout the grid segment will be expected.

A relationship between different data samples collected within a gridsegment can be analyzed in a co-kriging procedure. A relationshipbetween data collected from (or otherwise associated with) multiple gridsegments can also be analyzed in a co-kriging procedure. In someembodiments, the difference in physical distance between the collecteddata samples is included in network quality calculations. In the same ordifferent embodiments, the difference between network parameters orfactors such as those of Table 2 is included in the network qualitycalculations. For example, some network quality parameters can becalculated using factors such as a distance between collection points ofdata. Other network quality parameters can be calculated using factorssuch as a difference in population in an area where one sample iscollected and where another sample is collected. In fact, the factorslisted in Table 2 can each be weighted accordingly in order to provide aknown component of interest about a particular grid segment, even whenvery little or no data has been collected within the grid segment, orwhen the data from the segment is aged. Furthermore, the factors can bearranged with varying degrees of influence upon each other. Stateddifferently, a co-kriging procedure can be used with estimates of thefactors in Table 2 to help interpolate a more precise estimate of thepCS.

FIG. 5D illustrates a group of graphs or plots correlating variousfactors that influence network quality in a given geographic area. Theplots represent exemplary and non-limiting mechanisms of data collectionand combination. Generally speaking, in the graphs of FIG. 5D, thevertical axis represents a particular “factor of difference” betweendata samples that have been collected. The factor of difference mayundergo statistical normalization procedures such as squaring,averaging, etc. The factor of difference may be measured in decibels(e.g., signal to noise), watts (e.g., signal strength), a unit count(e.g., number of people, number of buildings, etc), units of time (e.g.,days, hours, etc.), and the like. The horizontal axis of the graphs inFIG. 5D may be units of distance (e.g., feet, miles, etc. as applied tolinear distance or altitude, etc.), density (e.g., foliage coverage,population, building height or development of acreage, etc.), and thelike. The points plotted may include collected data samples, interimcalculation terms, frequencies, data rates, and many other points. Othergraphs, which is to say other collections and amalgamations of data, mayof course be formed.

The graphs of FIG. 5D include “model lines.” The model lines generallyrepresent information variances between collected data samples. As themodel lines are plotted or otherwise analyzed, certain trends can becomeapparent, and a consistent measure of weighting can be applied to adesirable level of accuracy. Although outlying data that is clearly inerror can be discarded, other outlying data is simply factored in, andthe influence that the outlying data exerts on more relevant data isreduced.

In some cases, which are particularly evident in the graphs having ahorizontal axis representing distance, a model line does not go throughthe origin. In cases such as this, it is recognized that situationsexist where sufficient data is collected, but the network qualityparameter nevertheless includes a random component that reduces thereliability of predicted results. In other cases, it is recognized thatsome plots will include points with widely varying values for the factorof difference (vertical axis). In these cases, a model line canbasically be drawn horizontally, therein indicating that no weightingtechnique is going to be of use because no discernible spatialrelationship exists between the collected samples.

In many cases, the graphs of FIG. 5D may be considered variograms (orsemivariograms). That is, plots used in geostatistical analysis toidentify variances between data points as a function of distance. Fromthe generated plots, insight can be drawn by keeping track of multiplevalues and analyzing them as a function of geographical location ordistance. In particular, a weighted mean can be calculated for theparticular factor of difference that is being plotted. In some cases,the weighted mean can then be used to predict a corresponding factor ofdifference in other grid segments, even in grid segments having nousable collected data.

One measure of the reliability of the weighted mean factors ofdifference is a calculated “standard error of the weighted mean.” Thestandard error of the weighted mean is a confidence level in a networkquality measurement for a grid segment (or a group of grid segments)wherein the network quality measurement is derived from related samplesof data. Stated differently, the “standard error” is, in effect, ameasurement of the difference between an estimator value and the truevalue being estimated.

A desirable estimator (i.e., weighting factor) for the predictivecomposite score (pCS) is one that is acceptably reliable. An acceptablyreliable estimator has a reduced standard error. To reduce the standarderror, the weights accorded to each sample can be adjusted.

Mathematically, using the kriging algorithms alone or in cooperationwith co-kriging algorithms, the sampled data can be transformed andplotted as in the semivariograms of FIG. 5D. From the plots, a set ofsimultaneous equations can be analyzed for solutions that yield anacceptably low standard error. The solutions include the weightingfactors that are applied in the generation of the pCS.

FIG. 5E illustrates a set of exemplary results of kriging algorithmsaccording to one embodiment. The illustration of FIG. 5E is an expandedview of certain ones of the grid segments of FIG. 5C. In each of thegrid segments, the geographic points where actual network performancedata was collected are also illustrated. FIG. 5E shows an exemplaryresult of a kriging procedure used on collected sample data.

In FIG. 5E, each grid segment is assigned an estimated pCS value and acorresponding standard mean error. The pCS values of FIG. 5E may bederived in a manner similar to the Composite Score values described, forexample, in Equation (4). Alternatively, the pCS values of FIG. 5E maybe formed to represent other network quality parameters.

In the embodiment of FIG. 5E, which is exemplary, a 95% confidence levelhas been estimated by a model fitting procedure used to predict the pCSvalue from the actually collected samples. That is, the sample datacollected within the grid segments includes results of actually measuredkey performance indicator network parameters. The known data can be usedin conjunction with a kriging and co-kriging algorithm to fit modelparameters by adjusting certain weighting values of the Table 2 factors.By dynamically adjusting the weighting factors, a 95% confidence levelcan be estimated for each pCS value for each grid segment.

Looking at the resultant data in FIG. 5E, each grid segment includes apredicted composite score (pCS) and a standard mean error for the pCS ofthe grid segment. For example, in grid segment 8-E a pCS value of 3.8has been calculated. There is a 95% confidence level that the actual pCSvalue is between 3.3 and 4.3. In grid segment 8-F, there is a 95%confidence level that the pCS value is between 3.8 and 4.2. Generallyspeaking, it is expected that grid segments having more collected datasamples will have a tighter range of expected pCS values, andconversely, grid segments with no data or old data will have a widerrange of expected pCS values, but this is not always the case. Forexample, the grid segment 9-F has no collected data, but a tight pCSrange between 3.5 and 3.9. One reason for this, for example, may be avery predictable geographic topology with known cellular infrastructure.Other non-linear results can also be noticed.

The non-linear pCS results across various grid segments in FIG. 5E comesas a result of the Table 2 factors used to weight the calculations madein the kriging algorithms. For example, a high building density in agiven segment may serve to lower (or widen) the range of a predictablecomposite score. Alternatively, a sparsely populated area having aconsistent topology and altitude may permit wireless network signals toeasily and reliably travel significant distances. Accordingly, anaccurate pCS value can be calculated within a predictably tight range.

In addition to the kriging and co-kriging techniques, other techniquescan also be applied to generate various pCS values. One techniqueincludes spatio-temporal averaging. In some embodiments, thespatio-temporal averaging techniques are performed to prepare data foruse within the kriging techniques.

Generally speaking, spatio-temporal averaging includes averaging a setof collected data over geographic space and/or over time. The averageddata can be used in the predictive composite score calculations. WhenKPI data related to a wireless network is collected, the data isrecognized as being accurate to the geographic location at the time thedata is collected. The collected data can provide useful informationabout nearby geographic locations. Additionally, the collected data ismore relevant to pCS calculations the sooner it is used. As time passes,the collected data grows stale, and the usefulness of the data candiminish.

In some embodiments, data is collected and spatially averaged within agiven grid segment. In other embodiments, data is collected andspatially averaged across several grid segments. The spatial averagingmay include simple averaging, wherein a sum of values is divided by thenumber of samples. The spatial averaging may also include weightedaveraging. In one embodiment of weighted averaging, data collected in agrid segment is fully weighted, data collected in immediately adjacentsegments is partially weighted at a first level, and data collected atnearby, non-adjacent grid segments is partially weighted at a secondlevel, lower than the first level. The weighted data samples are summedin the embodiment and the sum is divided by the number of samples. Otherweighting techniques are also possible.

Some embodiments collect and temporally average KPI network data. It isrecognized that as time passes, certain data (e.g., signal strength,SNR, etc.) becomes less relevant to pCS calculations. Accordingly,techniques can be employed to reduce the influence of some data as thedata ages. In one embodiment, a weighting factor applied to certain datais reduced by ten percent per hour for a determined number of hours. Theweighting factor may subsequently be reduced further as more hours pass,but the rate of reduction of the weighting factor can be changed. Forexample, it has been recognized in some embodiments that signal strengthis very relevant in the first few hours after the signal strength datahas been collected. Accordingly, in the embodiment, the signal strengthdata can be fully weighted for the first four hours. Subsequently, everyfour hours thereafter, the weighting of the signal strength data bereduced by some amount until the weighting reaches a desired level.Later still, the weighting factor value can be reduced further as morehours or days pass. After a known amount of time has passed, theweighted data may have little or no influence on pCS calculations.

Spatio-temporal averaging combines spatial and temporal averaging of thecollected KPI data. The averaging calculations are performed on raw datain some cases, and in other cases, the data is mathematically adjustedand/or combined before being averaged.

Other techniques applied to generate various pCS values include datamining techniques. Data mining techniques include both classificationalgorithms and regression algorithms. Both classification techniques andregression techniques are often executed as tree based algorithms, thusthey may be known as Classification and Regression Tree (CART)techniques. In the CART techniques, analysis includes one or moreprogressive or recursive sequences of binary (e.g., if-then) decisions.In some embodiments, the data mining algorithms are performed to preparedata for use within the kriging algorithms.

Generally speaking, classification techniques iteratively process datato continuously predict categorical variables, and the regressiontechniques iteratively process data to predict continuous variables.Known CART techniques include “bagging,” “random forests,” and others.

Regression tree techniques are employed to predict the values of acontinuous variable from one or more continuous and/or categoricalpredictor variables. For example, in one embodiment, an expected signalstrength value within an identified grid segment is predicted. Theexpected signal strength value is a continuous dependent variable.Various continuous predictors (e.g., distance to a nearest cell tower,topology, percent development, etc.) as well as categorical predictorvariables (e.g., network technology, wireless carrier frequency, etc.)are used to create the prediction. The regression tree techniques can beapplied to create an algorithm wherein weighting factors are chosenbased on a sequence of decision-driven analyses of individual data.Additionally, similar techniques can also be applied to combinations ofdata.

Classification tree techniques are employed to predict the value of acategorical dependent variable (i.e., a variable dependent on the class,group membership, etc. of the data) from one or more continuous and/orcategorical predictor variables. For example, in some embodiments,predictions are made as to whether a call request initiated by a mobiledevice user will connect, whether an initiated call will be maintainedfor the next N seconds, whether an initiated call will be maintainedthrough a cell tower handoff. Predictions such as these and others areexamples of simple binary classification problems. That is, thecategorical dependent variable can assume only one of two distinct andmutually exclusive values. In other embodiments, predictions such aswhich tower will receive an already initiated call in a handoff will bemade. In these types of categorical dependent variable cases, aprediction is made as to which one of multiple category or classalternatives will be selected for the categorical dependent variable.The classification tree techniques can be applied when a range ofpossible outcomes for a single event or datum are differently weighted.

Exemplary CART methods employed in embodiments to produce pCS values arenonparametric and nonlinear. The resulting values from the tree-basedclassification and regression algorithms are derived from a generallysmall series of logical if-then conditions (tree nodes). The algorithmsdo not require implicit assumptions about the underlying relationshipsbetween the predictor variables and the dependent variable, and theactual relationships are not necessarily linear. In one example ofnon-linearity, a continuous outcome variable (e.g., likelihood ofmaintaining a call) could be positively related to an incoming variable(e.g., signal strength) if the incoming variable is greater than somecertain amount in a grid segment having a certain characteristic (e.g.,building density), but negatively related if the variable is the sameunder different characteristics or less than that amount under anycharacteristics. i.e., the tree algorithm can reveal multiple splitsbased on a value of a single incoming variable, thus illuminating anon-linear relationship between the variables.

CART methods are particularly well suited for data mining tasks wherethere is often little a priori knowledge nor any coherent set oftheories or predictions regarding which variables are related and how.In these types of data analyses embodiments, the CART algorithms areemployed to reveal simple relationships between just a few variablesthat are easily unnoticed using other analytic techniques.

Still other techniques can be used to generate pCS values. For example,model selection techniques can be used to improve the parametersselected for the pCS calculations. In more detail, model selectiontechniques can be used to select between different model forms (oftenwith a different number of parameters). The techniques can be used toavoid “over-fitting” of models and thereby avoid, for example,predicting noise instead of signal.

In the model selection techniques, various parameters can be set withcontrolled “model” data to mirror results observed from the applicationof actual, collected data. The model data can be adjusted for manyreasons, including to change how accurately the observable resultsmirror actual results. As models are developed, the model data can beapplied in predictive composite score algorithms where actual collecteddata is non-existent, stale, or otherwise limited.

As described herein, kriging and co-kriging techniques are applied inspatial interpolation to weight unknown or unreliable data by distance(and other factors such as listed in Table 2) to known data.Spatio-temporal averaging, data mining, and modeling techniques are alsoapplied to produce weighting data. The weighting is used to produce areliable predictions about areas (e.g., grid segments) having old data,unreliable data, or no data whatsoever.

The factors listed in Table 2 can be gathered within the wirelessnetwork performance measurement system of FIG. 2. Alternatively, or inaddition, certain data can be recorded in other systems and retrievedwhen needed for use in the kriging algorithms or other techniques togenerate pCS values. For example, known databases exist that includelandscape attributes, population density, percentage development (e.g.,concrete, building height, etc.), topography (e.g., altitude changebetween two points), and many other types of data.

In some embodiments, a calculated pCS value is an indication ofpredicted “signal quality.” The predicted signal quality is acombination of the impact of signal strength and other factors thatprovides a likelihood of making and maintaining a communicativeconnection in a given wireless data network.

In some embodiments, signal quality is a modification of signal strengthdata. Signal strength can be very different between carriers in a commongeographic location. Accordingly, the pCS values calculated for onenetwork carrier can be very different from those of another networkcarrier. Furthermore, a pCS value can also be created to include dataperformance, data speed, data reliability, access to data transfer, dataretention, dropped connections, packet loss, and many other factors.

One example embodiment of a predictive composite score (pCS) calculationis now presented. In the embodiment, data is collected from one or moremobile devices. The data may be specifically collected for the purposeof collecting data performance data or the data may be collected as acrowd-sourced process. The data generally includes signal strengthmeasurement data, network carrier data, time stamp data, and ageographic location indicating where the data was collected. Additionaldata may also be collected, for example, the results of data speed ordata integrity tests (e.g. download or upload speed) and the like. Inthe embodiment, the geographic location is recognized as being within aboundary of a particular grid segment block such as illustrated in FIGS.5C and 5E. Data for the grid segment is averaged and normalized, forexample with spatio-temporal and data mining techniques. Optionally,modeling techniques are used to replace or supplement data that is, forone reason or another, unavailable or unused. In some cases, known “bad”data is deleted or otherwise ignored. In some cases, outlying data ismaintained and used in the calculation of the pCS, and the outlying datahas reduced influence on a particular pCS calculation.

Subsequent to the data collection procedures, certain rules can beapplied as part of the pCS calculation. The rules can be weightingfactors that increase or reduce the influence of the data in the pCScalculation. Rules can be applied for one or more of the factors listedin Table 2. For example, in some areas, the geographic topologyincreases the influence of one network technology (e.g., CDMA) overanother network technology (e.g., GSM). In certain areas, the populationdensity, tower location, or other factors influence the pCS calculation.In subsequent steps, kriging and optionally co-kriging algorithms areapplied to reduce or increase the influence of data collected in onegrid segment on the pCS calculation of another grid segment. As a final,optional step, the pCS calculation for each particular grid segment isnormalized to within a certain desirable range, for example 1-5.

Additional pCS values can be similarly created to include measurementsbeyond signal quality. For example, certain pCS values are indicators ofpredicted data communications (e.g., speed and data throughput). In oneembodiment, a data communication quality pCS value is formed byaveraging 35% download, 15% upload, and 50% signal strength properties,and applying kriging algorithms to influence the value in each givengrid segment.

In another embodiment, one or more databases include training data sets.The training data is used to model expected pCS results against actualpCS calculations. The comparative analysis is used to update thetraining data.

In another embodiment, external databases are accessed to retrieve datathat can be layered into the pCS calculations during the krigingprocedures. For example, National Weather Service (NWS) data andNational Oceanic and Atmospheric Administration (NOAA) data may be used.National Aeronautics and Space Administration (NASA) data may be used.The data may be related to weather, solar activity, climate patterns,earth's magnetic field data, and many other things. Additionally, datafrom social network sites can be accessed to include trending or “buzz”data related to localized activity that can influence network carrierreliability, quality, or other user experience.

In another embodiment, website 150 provides some or all of the contentdescribed herein, but additionally or alternatively, wireless networkperformance software application 142 presents another interface foroutputting screen data, audio data, and other interactive or staticinformation. For example, a dashboard interface may be presented. Thedashboard, conventionally known to one skilled in the art, createsscreen data that represents a solid or semi-transparent window on adisplay to present particular information. In some cases, the dashboardis displayed on a local computer, and in other cases, the dashboard ispresented on a computer communicatively located within a private localarea network (LAN) or the like. That is, all interaction with website150 can be from a single computer or from a serving host computer and aremote user computer.

A dashboard interface, like a website, provides an interactive facilitywhereby a user can retrieve data, input control information, and/ordirect how data will be presented. The dashboard interface may be astandalone interface application or may be presented within anothersoftware application (e.g., a browser).

In still another embodiment, wireless network performance softwareapplication 142, directly or cooperatively with website 150, presentsnetwork performance data to another computer in a known or proprietaryformat. For example, the data used to populate website 150 may bestreamed. In cases where the data is streamed, one or more receivingcomputers are capable of parsing the data stream and processing theinformation presented within. The information often includes real timenetwork performance data in either a raw format or as processed by thewireless network performance software application 142.

In some cases, a receiving computer provides input to web server 126 todirect the streaming. The input may provide geographic or otherindicators that direct the wireless network performance softwareapplication 142 to provide data related to the requested indicators.

In still other embodiments, significant quantities of post-processeduser quality data are provided by the wireless network performancesoftware application 142. The post-processed data is generally retrievedfrom a local memory; however, the data in local memory is often derivedfrom raw data stored on server 120. The data provided may includeparticular KPIs, particular “best supplier” data, or the like. The datamay be in a predefined format so that the receiving device can rebuildall or part of a database or other memory structure maintained by thewireless network performance software application 142.

FIG. 6 is a flowchart 600 illustrating processes that may be used byembodiments of the web server used for the presentation networkperformance data. In this regard, each described process may represent amodule, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It should also be noted that in some implementations, thefunctions noted in the process may occur in a different order, mayinclude additional functions, may occur concurrently, and/or may beomitted.

With respect to FIG. 6, the process 600 is ongoing with operation of theweb server 126, and is illustrated as starting at 602 when web-server126 initializes. At 604, a database of raw or pre-processed user qualitywireless network data is accessed, and at 606, the data undergoesadditional processing, or post-processing. If it is time to update at608, then the process returns to retrieve more data to 604. If it is nottime to update, then at 610, a user input condition is interrogated. Ifuser input is found, the user-requested data is gathered and formattedat 612. If the formatted and/or post processed data is ready for outputat 614, then the formatted data is output via the requested medium at616. The process 600 does not typically end, but instead, cyclicallyreturns to receive and/or retrieve a new batch of raw or pre-processeddata at 604.

FIG. 6A is a flowchart 500 illustrating processes that may be used byembodiments of the mobile device used for collecting network performancedata. In this regard, each described process may represent a module,segment, or portion of code, which comprises one or more executableinstructions for implementing the specified logical function(s). Itshould also be noted that in some implementations, the functions notedin the process may occur in a different order, may include additionalfunctions, may occur concurrently, and/or may be omitted.

With respect to FIG. 6A, the process 500 is ongoing with operation ofthe mobile device 118 a-d, and is illustrated as starting at 502 whenthe mobile device 118 a-d initializes. At 504, operation of the mobiledevice 118 a-d begins in a Service Mode. If a configuration file is tobe loaded at 506, then the new configuration file is stored at 508. Whenthe mobile device 118 a-d is directed to collect data at 510, a set ofone or more KPI data are collected at 512. The process determines if aserver 120 is to be updated with collected KPI data at 514, and if so,the update is performed at 516. If the Application Mode is to be invokedat 518, then the Application Mode presents user choices at 520. Theprocess 500 does not typically end, but instead, cyclically returns tothe Service Mode operation at 504.

FIG. 6B is a flowchart 400 illustrating processes that may be used byembodiments of the server used for directing the collection andretrieval of network performance data. In this regard, each describedprocess may represent a module, segment, or portion of code, whichcomprises one or more executable instructions for implementing thespecified logical function(s). It should also be noted that in someimplementations, the functions noted in the process may occur in adifferent order, may include additional functions, may occurconcurrently, and/or may be omitted.

With respect to FIG. 6B, the process 400 is ongoing with operation ofthe server 120, and is illustrated as starting at 402 when server 120initializes. At 404, operation of server 120 begins by creating a newdatabase or mounting an existing database. Next, at 406, the number andidentity of mobile devices 118 a-d already in the network performancemeasurement system 110 are determined. If there are new mobile devices118 a-d in the network performance measurement system 110 at 408, thenconfiguration data (e.g., a configuration file) is created and arelationship is established between server 120 and each new mobiledevice 118 a-d at 410.

At 412, server 120 determines if it is time to send configuration datato mobile devices 118 a-d or if mobile devices have requested newconfiguration data, and if so, the updated configuration data is sent tothe mobile devices 118 a-d. Next, at 416, server 120 determines if datais to be collected from mobile devices 118 a-d, and if so, the data iscollected at 418.

Finally, server 120 determines if the collected user quality datameasurements are to be disseminated at 420. If the data is to bedisseminated, the collected data is communicated to another server orapplication at 422. The process 400 does not typically end, but instead,cyclically returns to the new devices interrogation operation at 408.

FIG. 6C is a flowchart 600 illustrating processes that may be used byembodiments of the wireless network performance measurement system 110(FIG. 3) to generate a predictive composite score. The operations may beconducted in a server, such as server 120 (FIG. 3). In this regard, eachdescribed process may represent a module, segment, or portion ofsoftware code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat in some implementations, the functions noted in the process mayoccur in a different order, may include additional functions, may occurconcurrently, and/or may be omitted.

With respect to FIG. 6C, the process 600 is ongoing with operation ofthe system 110, and is illustrated as starting at 602 when a server 120initializes. At 604, key performance indicator (KPI) data related to awireless network is collected. In some embodiments, the flowchart ofFIG. 6A describes and illustrates a KPI collection process for a mobiledevice, and the flowchart of FIG. 6B describes and illustrates a serverprocess that facilitates the KPI data collection.

At 606, data from spatial and a-spatial sources (e.g., as illustrated inFIG. 6C, as expressed in Table 2, etc.) is combined. Optionally,training data is also included. The spatial and other data can be usedto produce weighting factors, which are applied to certain ones of theKPI data. At 608, geostatistical methods produce interim predictiveweighting data that can be applied to KPI values. The predicted valuesare combined over time to create averages that are both spatial andtemporal at 608. At 610, spatial attributes are used uniformly or inlayers along with weighted, averaged KPI values to create a predictivecomposite score (pCS). The process cyclically returns to 604 whereadditional KPI data is collected, new predictive scores are calculated,and interim predictive data is updated.

The various embodiments described above can be combined to providefurther embodiments. All of the U.S. patents, U.S. patent applicationpublications, U.S. patent applications, foreign patents, foreign patentapplications and non-patent publications referred to in thisspecification and/or listed in the Application Data Sheet areincorporated herein by reference, in their entirety. Aspects of theembodiments can be modified, if necessary to employ concepts of thevarious patents, applications and publications to provide yet furtherembodiments.

These and other changes can be made to the embodiments in light of theabove-detailed description. In general, in the following claims, theterms used should not be construed to limit the claims to the specificembodiments disclosed in the specification and the claims, but should beconstrued to include all possible embodiments along with the full scopeof equivalents to which such claims are entitled. Accordingly, theclaims are not limited by the disclosure.

1. A method to form a predictive confidence level, comprising:retrieving geographically based wireless networking performance datafrom a first computing device, the geographically based wirelessnetworking performance data derived from at least one mobile device;spatially averaging at least some of the geographically based wirelessnetworking performance data; temporally averaging at least some of thegeographically based wireless networking performance data; andgeostatistically combining the temporally and spatially averagedgeographically based wireless networking performance data to form apredictive confidence level, the predictive confidence level being aforecast of expected success of a determined wireless communicationnetwork event in the first geographic region.
 2. The method to form apredictive confidence level of claim 1 wherein the predictive confidencelevel is an indication of wireless network quality that can be expectedin an identified geographic area.
 3. The method to form a predictiveconfidence level of claim 2 wherein the identified geographic area isselectable by a user.
 4. The method to form a predictive confidencelevel of claim 3 wherein the predictive confidence level is recalculatedfor a second identified geographic area selected by a user, the secondidentified geographic area having a different size or shape from theidentified geographic area.
 5. The method to form a predictiveconfidence level of claim 1 wherein the determined wirelesscommunication network event is a connection of a wireless call.
 6. Themethod to form a predictive confidence level of claim 1 wherein thedetermined wireless communication network event is a maintaining of awireless call through one or more handoffs.
 7. The method to form apredictive confidence level of claim 1 wherein the determined wirelesscommunication network event is a certain rate of data transfer over awireless communication network.
 8. The method to form a predictiveconfidence level of claim 1 wherein the predictive confidence level isformed in association with a first operator of a wireless communicationnetwork and a second predictive confidence level is formed inassociation with a second operator of a second wireless communicationnetwork.
 9. The method to form a predictive confidence level of claim 1,wherein the act of spatially averaging at least some of thegeographically based wireless networking performance data comprises:creating a plurality of weighting factors to increase or decrease theinfluence of the at least some of the geographically based wirelessnetworking performance data, each of the plurality of weighting factorsbased on a different network quality influence factor; and applying oneor more of the plurality of weighting factors to individual ones of theat least some of the geographically based wireless networkingperformance data.
 10. The method to form a predictive confidence levelof claim 9 wherein a network quality influence factor is valueassociated with: a degradation of carrier signal over terrain, adistance between a tower and a mobile device, a distance between towers,a tower capacity, a terrain foliage attribute, a terrain topologyattribute, population density, a time of day, a time of year, a weatherevent, a traffic pattern, or a network technology.
 11. A method to formand present a predictive confidence level, comprising: assigningboundaries to a plurality of geographic areas; retrieving from at leastone memory device a set of wireless networking performance dataassociated with at least one of the plurality of geographic areas, theset of wireless networking performance data derived from at least onemobile device; temporally averaging at least some of the wirelessnetworking performance data from the retrieved set of wirelessnetworking performance data; spatially aggregating at least some of thewireless networking performance data from the retrieved set of wirelessnetworking performance data; geostatistically calculating a predictiveconfidence level from the spatially aggregated data, the predictiveconfidence level being a forecast of expected success of a determinedwireless communication network event in the at least one of theplurality of geographic areas; and presenting the predictive confidencelevel to an output port.
 12. The method to form and present a predictiveconfidence level of claim 11 wherein spatially aggregating at least someof the wireless networking performance data includes executing a krigingalgorithm.
 13. The method to form and present a predictive confidencelevel of claim 11 wherein spatially aggregating at least some of thewireless networking performance data includes averaging a set ofcollected data over geographic space and over time, wherein the set ofcollected data is weighted based on a distance between a geographicpoint of interest and a geographic area where each datum of the set ofcollected data was collected, and wherein each datum of the set ofcollected data is weighted based on a time that the datum was collected.13. The method to form and present a predictive confidence level ofclaim 11 wherein the determined wireless communication network eventestimates a confidence with which a wireless call can be successfullyconnected.
 14. The method to form and present a predictive confidencelevel of claim 11 wherein the determined wireless communication networkevent estimates a confidence with which a wireless call will remainconnected through one or more handoffs.
 15. The method to form andpresent a predictive confidence level of claim 11 wherein the determinedwireless communication network event estimates a confidence with which adata rate above a selected threshold can be achieved.
 16. Anon-transitory computer readable medium comprising content thatconfigures a computing device to perform acts, comprising: retrieving aset of geographically based wireless networking performance data derivedfrom a plurality of mobile computing devices; spatially averaging atleast some of the geographically based wireless networking performancedata; temporally averaging at least some of the geographically basedwireless networking performance data; and geostatistically combining thetemporally and spatially averaged geographically based wirelessnetworking performance data to form a predictive confidence level, thepredictive confidence level being a forecast of expected success of anevent occurring at least partially in a first geographic region across awireless communication network.
 17. The non-transitory computer readablemedium of claim 16 wherein the content configures the computing deviceto perform acts, wherein the event occurring at least partially in thefirst geographic region across the wireless communication network is aconnection of a wireless call.
 18. The non-transitory computer readablemedium of claim 16 wherein the content configures the computing deviceto perform acts, wherein the event occurring at least partially in thefirst geographic region across the wireless communication network is acertain rate of data transfer over the wireless communication network.19. The non-transitory computer readable medium of claim 16 wherein thecontent configures the computing device to perform acts, whereingeostatistically combining temporally and spatially averagedgeographically based wireless networking performance data includesexecuting a kriging algorithm.
 20. The non-transitory computer readablemedium of claim 19 wherein the content configures the computing deviceto perform acts, wherein executing the kriging algorithm includesapplying at least one weighting factor based on a distance betweengeographic points where the geographically based wireless networkingperformance data has been collected.